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Preface 


This year we organized the 19th edition of the International Conference on Smart Living 
and Public Health (ICOST 2022), a series which has succeeded in bringing together a 
community from different continents for almost 20 years and raised awareness about 
frail and dependent people’s quality of life in our societies. 

After 18 very successful conferences held in France (2003, 2009, 2017), Singapore 
(2004, 2013, 2018), Canada (2005, 2011), Northern Ireland (2006), Japan (2007), the 
USA (2008, 2014, 2019), South Korea (2010), Italy (2012), Switzerland (2015), China 
(2016), and Tunisia (2020), this 19th edition of the conference was organized by the 
Institut Mines-Télécom (IMT), Paris, France, and held during June 27-28, 2022. The 
conference was organized as an official event of the French Presidency of the Council 
of the European Union. The theme of the conference was “Participative Urban Health 
and Ageing Well in the Age of AI.” 

ICOST 2022 provided a premier venue for the presentation and discussion of research 
on the design, development, deployment, and evaluation of AI for health, smart urban 
environments, assistive technologies, chronic disease management, and coaching and 
health telematics systems. ICOST 2022 aimed to understand and assess the diverse and 
disparate impact of digital technologies on public health in developing and developed 
countries. It brought together stakeholders from health care, public health, academia, and 
industry along with end users and family caregivers to explore how to utilize technologies 
to foster health, disease prevention, and independent living and offer an enhanced quality 
of life. The ICOST 2022 conference featured a dynamic program incorporating a range 
of oral and poster presentations along with panel sessions. 

ICOST 2022 was proud to extend its hospitality to an international community 
consisting of researchers from major universities and research centers, representatives 
from industry, and users from 17 different countries. We would like to thank the authors 
for submitting their current research work and the Program Committee members for 
their commitment to reviewing submitted papers. The ICOST proceedings and chapters 
have now reached over 400,000 downloads and are in the top 25% of downloads of 
Springer LNCS. We are extremely thankful to our sponsors for their commitment and 
support to the vision and mission of ICOST. 


June 2022 Hamdi Aloulou 
Bessam Abdulrazak 

Antoine de Marassé-Enouf 

Mounir Mokhtari 


Organization 


General Chair 


Mounir Mokhtari Institut Mines-Télécom, France 


General Co-chair 


Antoine de Marassé-Enouf Agence du Numérique en Santé (French eHealth 
Agency), France 


Scientific Advisory Board 


Mounir Mokhtari Institut Mines-Télécom, France 

Bessam Abdulrazak University of Sherbrooke, Canada 

Sumi Helal University of Florida, USA 

Jose Pagan New York University and the New York Academy 
of Medicine, USA 

Maria Teresa Arredondo Universidad Politécnica de Madrid, Spain 

Maria Fernanda Cabrera University Politecnica de Madrid, Spain 

Mohamed Jmaiel Digital Research Center of Sfax, Tunisia 

Dong Jin Song National University of Singapore, Singapore, and 
Griffith University, Australia 

Hamdi Aloulou University of Monastir and Digital Research 


Centre of Sfax, Tunisia/Institut 
Mines-Télécom, France 


Liu Yang Nanyang Technological University, Singapore 
Jongbae Kim Yonsei University, Korea 

Timo Jámsá University of Oulu, Finland 

Sungyoung Lee Kyung Hee University, South Korea 

Hisato Kobayashi Hosei University, Japan 

Daby Sow IBM Research AI, USA 

Daging Zhang Télécom SudParis, France 


Program Committee Chairs 


Bessam Abdulrazak University of Sherbrooke, Canada 

Hamdi Aloulou University of Monastir and Digital Research 
Centre of Sfax, Tunisia/Institut 
Mines-Télécom, France 


viii Organization 
Program Committee 


Afef Mdhaffar 

Aitor Almeida 

Armel Ayimdji Tekemetieu 
Belkacem Chikhaoui 
Boussada Rihab 

David Menga 

Elyes Lamine 

Eric Campo 


Hassan Mostafa Ahmed Fahmi 
Hisato Kobayashi 

Hongbo Ni 

Houssem Aloulou 

Hui Chen 

Jean Marie Bonnin 

Jeffrey Soar 

Jéróme Boudy 


Laurent Billonnet 
Laurent Clavier 
Manfred Wojciechowski 


Meriem Zerkouk 


Salim Hima 
Sergio Copelli 
Stefanos Kollias 
Valérie Gay 
Vigouroux Nadine 


Wael Sellami 
Xiaolan Xie 
Zuraimi Sultan 


Organizers 


Institut Mines Télécom, France 
Université de Sherbrooke, Canada 


University of Sfax, Tunisia 

University of Deusto, Bilbao, Spain 

Université de Sherbrooke, Canada 

University of Quebec, Canada 

University of Manouba, Tunisia 

EDF R&D, France 

IMT Mines Albi and ISIS Castres, France 

LAAS-CNRS, Université Toulouse-Jean Jaurés, 
France 

Université de Sherbrooke, Canada 

Hosei University, Japan 

Northwestern Polytechnical University, China 

University of Sfax, Tunisia 

Université de Sherbrooke, Canada 

IMT Atlantique, France 

University of Southern Queensland, Australia 

Télécom SudParis and Institut Polytechnique de 

Paris, France 

University of Limoges, France 

IMT Nord Europe, France 

University of Applied Sciences Dusseldorf, 

Germany 

University of Science and Technology of Oran, 
Algeria 

ESME-Sudria, France 

MultiMed Engineers srls, Italy 

University of Lincoln, UK 

University of Technology Sydney, Australia 

Institut de Recherche en Informatique de 
Toulouse, France 

University of Sfax, Tunisia 

École des Mines de Saint Étienne, France 

Berkeley Education Alliance for Research in 
Singapore (BEARS), Singapore 


Digital Research Center of Sfax, Tunisia 
National University of Singapore, Singapore 


Nanyang Technological University, Singapore 
New York University, USA 


Sponsor 


Agence du Numérique en Santé (French eHealth Agency), France 


Organization 


Contents 


IoT and AI solutions for E-health 


Self-healing Approach for IoT Architecture: AMI Platform .................. 3 
Bessam Abdulrazak, Josué Ayi Codjo, and Suvrojoti Paul 


Digital Twin Driven Smart Home: A Feasibility Study ...................... 18 
Alireza Asvadi, Andrei Mitriakov, Christophe Lohr, 
and Panagiotis Papadakis 


Modeling IoT Design Patterns Proven Correct by Construction ............... 30 
Imen Tounsi, Najeh Khalfi, Abdessamad Saidi, and Mohamed Hadj Kacem 


IoT Architecture with Plug and Play for Fast Deployment and System 

Reliability: AMI Platform  Leossssenéerr Rr RR tibi AA N ERR RR ERES 43 
Bessam Abdulrazak, Suvrojoti Paul, Souhail Maraoui, Amin Rezaei, 
and Tianqi Xiao 


Annotation Systems in the Medical Domain: A Literature Review ............ 58 
Zayneb Mannai, Anis Kalboussi, and Ahmed Hadj Kacem 


Wellbeing Technology 


SAATHI: An Urdu Virtual Assistant for Elderly Aging in Place .............. 73 
Anand Kumar, Ghani Haider, Maheen Khan, Rida Zahid Khan, 
and Syeda Saleha Raza 


Smart Technology in the Home for People Living in the Community 

with Mental Illness and Physical Comorbidities ......................000005 86 
Cheryl Forchuk, Abraham Rudnick, Deborah Corring, Daniel Lizotte, 
Jeffrey S. Hoch, Richard Booth, Barbara Frampton, Rupinder Mann, 
and Jonathan Serrato 


Toward a Trip Planner Adapted to Older Adults Context: Mobilainés Project ... 100 
Bessam Abdulrazak, Sahar Tahir, Souhail Maraoui, 
Véronique Provencher, and Dany Baillargeon 


Data-Driven Smart Medical Rehabilitation Exercise and Sports Program 

Using a Living Lab Platform to Promote Community Participation 

of Individuals with a Disability: A Research and Development Pilot 

POOH Lii deLIPe AT QquiS PELLIS ASSI UC iocis AS chee Chee peek Ee 112 
Seungbok Lee, Yim-Taek Oh, Hogene Kim, and Jongbae Kim 


xii Contents 


Real-Time Human Activity Recognition in Smart Home on Embedded 
Equipment: New Challenges cscs esicee sev Rr RR RE RR RYE 125 
Houda Najeh, Christophe Lohr, and Benoit Leduc 


E-health Solutions for COVID 19 


Design COVID-19 Ontology: A Healthcare and Safety Perspective ........... 141 
Hamid Mcheick, Youmna Nasser, Farah Al Wardani, and Batoul Msheik 


Social Response to COVID-19 SMART Dashboard: Proposal for Case 

Mn A —————————— nnn! 154 
Karenina Zaballa, Gabriela Fernandez, Carol Maione, 
Norbert Bonnici, Jarai Carter, Domenico Vito, and Ming-Hsiang Tsou 


Adopting the Internet of Things Technology to Remotely Monitor 

CONIDSTS Faen des ede uber ERERPEQEEOREP SOR C PPbEt OD Ei PE S pops 166 
Abdessamad Saidi, Mohamed Hadj Kacem, Imen Tounsi, 
and Ahmed Hadj Kacem 


Biomedical and Health Informatics 


Tree-Based Models for Pain Detection from Biomedical Signals ............. 183 
Heng Shi, Belkacem Chikhaoui, and Shengrui Wang 


Stress Prediction Using Per-Activity Biometric Data to Improve QoL 

it he Elderly AD """"""—"———————á 196 
Kanta Matsumoto, Tomokazu Matsui, Hirohiko Suwa, 
and Keiichi Yasumoto 


Short Contributions: Medical Systems and E-health Solutions 


An Exploratory Study on Development Smart Cradle for Women 

with Spinal Cord Injury: Focus Group Interview .................00e eee e es 241 
Jae-nam Kim, Ha-yeon Yang, Min-kyung Kim, Hyun-kyung Kim, 
Sun-hwa Shim, Eun-joo Kim, Wan-ho Jang, and Sun-young Jo 


ICT-Based Customized Off-Loading Cushion to Prevent Pressure Ulcers 
for People with Spinal Cord Injury: A Pilot Study ......................00.. 217 
Yun-hwan Lee, Kwang-tae Moon, Dong-wan Kim, and Jongbae Kim 


Autism Spectrum Disorder (ASD) Detection Using Machine Learning 
FIPS: Grodad-rbnee s ook UTERE PES EPOR EUR U EDD Did ébai Ki edi $uis 225 
Naouel Boughattas and Hanen Jabnoun 


Contents xiii 


Ant Colony Optimization with BrainSeg3D Protocol for Multiple Sclerosis 
Lesion Detecülofli 22 sex shee ad Re tole Pr ERU RP eR er ed a 234 
Dalenda Bouzidi, Fahmi Ghozzi, and Ahmed Fakhfakh 


A Systematic Review on the Development of Clothing for People 

yo Disabil Nm OPER Sue eelrpip*IXeAcRgaReidkxBadepeadQ end P EdA 246 
Ha-yeon Yang, Hyun-kyung Kim, Min-kyung Kim, Sun-hwa Shim, 
Eun-ju Kim, Jae-nam Kim, Sun-young Jo, and Wan-ho Jang 


Short Contributions: Wellbeing Technology 


Empowering Well-Being Through Conversational Coaching for Active 

and Healthy Ageing .iooscecoscecetoetek o OR REOR EXER ERE HR d eR E d 237 
Michael McTear, Kristiina Jokinen, Mohnish Dubey, Gérard Chollet, 
Jérôme Boudy, Christophe Lohr, Sonja Dana Roelen, Wanja Mossing, 
and Rainer Wieching 


Smart Home-Based Home Modification Program for Persons 
vob Disabilities: A Pilot SEO uu esa duteseriirexibsertan ehe pola dol 266 
KwangTae Moon, YunHwan Lee, Dongwan Kim, and Jongbae Kim 


Mask Detection Using IoT - A Comparative Study of Various Learning 

lbi er "eer 212 
Mohamed Amine Meddaoui, Mohammed Erritali, Youness Madani, 
and Frangoise Sailhan 


Understanding the Knowledge, Perception and Uptake of Contraception 
in Nigeria: A Case Study of Saye-Zaria .....ieeseieeseesses eer nn 284 
Ayandunmola Folake Oyegoke and Aisha Abubakar 


In-Air Handwriting Recognition Using Acoustic Impulse Signals ............. 293 
Kai Niu, Fusang Zhang, Xiaolai Fu, and Beihong Jin 


Novel Interactive BRAINTEASER Tools for Amyotrophic Lateral 

Sclerosis (ALS) and Multiple Sclerosis (MS) Management .................. 302 
Sergio Gonzalez-Martinez, María Fernanda Cabrera-Umpiérrez, 
Manuel Ottaviano, Vladimir Urošević, Nikola Vojičić, 
Stefan Spasojević, and Ognjen Milićević 


PUNO UNGER E E E N eee babes ENEE TENE E E N E E Ed 311 


IoT and AI solutions for E-health 


® 


Check for 
updates 


Self-healing Approach for IoT Architecture: 
AMI Platform 


Bessam Abdulrazak, Josué Ayi Codjo, and Suvrojoti Pau? 2 


AMI-Lab, Dept. Infromatique, Faculty of Sciences, Université de Sherbrooke Research Center 
On Aging, Sherbooke, Canada 
Bessam.Abdulrazak,Josue.Ayi.Codjo, 


A 


Suvrojoti.Paul}@usherbrooke.ca 


Abstract. The fast growth of the IoT and the unlimited possibilities in terms of 
applications and processing brought forth by the 5G which is around the corner is 
making IoT an active part of the activity of daily living. Those massive architec- 
tures become then the target of security issues, broken services, broken hardware 
or malfunctioning of applications. Moreover, in a system of million connected 
devices, operating each one of them is impossible making the platform unmain- 
tainable. In this study, we present our attempt to achieve the autonomy of IoT 
infrastructure and we present some of the existing and recurrent issues under- 
mining the IoT architecture. Then we review existing self-healing techniques that 
enable a system to be autonomous and solve issues. We also described our IoT 
platform that targets the self-healing concern. Finally, we point out some recom- 
mendations to make an overall reliable, resilient loT system with cognitive entities 
for self-management. 


Keywords: IoT - Responsibility - Self-healing - Automation - Reliability - 
Networking 


1 Introduction 


An extensive growth in the development of the Internet of Things (IoT) in recent times 
has enabled the possibility of gathering a large amount of data from every aspect of 
life. This rapid growth in every sector imposes several challenging operational issues. 
It is possible to deal with these issues for a limited number of IoT devices however 
extremely difficult in case of a large number of devices. We argue that the self-healing 
approach can efficiently handle operational issues generated when various IoT devices 
are incorporated into an architecture. 

The global IoT market size is forecasted to grow around 1.6 trillion by 2025 [1]. These 
millions of connected devices create a massive amount of data and unlimited possibili- 
ties. Foreseeing that infinite connection allows IoT to bring its benefits in every aspect 
of life (e.g., in domains such as healthcare and industrialization). IoT-based systems 
have an important role in improving health-related quality of life. Still, they introduce 
various issues that require immediate attention from developers before considering a 
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large-scale deployment. IoT solutions, e.g., used in activity trackers, and sleep trackers 
aim at abstracting daily routines to offer better services. Albeit such solutions promote 
people to improve their typical daily activities, they are recurrently exposed to security 
breaches due to exposing acquired data to online services [2]. The IoT security aspect in 
today’s world is the number one operational issue addressed in literature [3]. Researchers 
proposed various self-protection solutions as countermeasures for reverting a system 
to a normal state [3]. Still, managing and handling a large number of devices impose 
challenging operational issues, e.g., energy depletion (battery saving), broken communi- 
cation (network connectivity), operating environment effects (overheating, cybercrime, 
storage management), discontinuity in services. Thus, these issues need to be addressed 
for every single device within an IoT infrastructure. Self-healing is an optimal app- 
roach for mitigating such issues with the ultimate goal to build reliable systems. It is 
related to a set of concepts (e.g., resilient systems [4], cognitive systems [5], responsible 
and autonomous systems [6]). It is a property of the system to identify and diagnose 
(i.e., independently) breakdowns and to autonomously determine and implement proper 
mitigation strategies. More specifically, self-healing confers system reliability through 
responsibility and awareness of the environment, issues, and expected countermeasures 
during unanticipated situations. Therefore, a self-healed IoT system should incorporate 
monitoring, awareness, and knowledge for detecting unforeseen states. Upon detecting a 
conceivable issue, the system initiates optimized planning for executing proper actions. 

The contribution of this study is twofold. First, highlighting the self-healing concerns 
of IoT solutions. Second, providing an insight into how a self-healing system should 
behave. We strive to achieve a self-healing system capable of reasoning over security, 
software, hardware, networking concerns and reacting in a proper way to keep the overall 
system on tracks without much human intervention. We consider, in our research, the 
capability of the system to sense, be aware of its environment and take proper actions 
for overcoming a problem. 

The rest of the paper is organized as follows: Sect. 2 introduces the IoT architecture 
andexisting issues. Section 3, present existing self-healing solutions. Section 4 depicts an 
IoT based case study. Section 5 describes the results of the experiments and evaluations 
conducted in order to validate the functionality of the system. Section 6 highlights our 
discussions for a reliable system. We finally outline the conclusion in Sect. 7. 


2 IoT Architecture and Existing Issues 


We introduce in this section an IoT architecture to better illustrate the operational issues 
and the required intervention at diverse levels (based on our AMI-Architecture [7]) 
(Fig. 1). * The Edge (or Device) layer (i.e., the closest layer to end-users/devices) consists 
of several devices, e.g., sensors, actuators, and smart devices. Geographically distributed, 
those devices are responsible for sensing environmental information and the edge layer 
is tasked with acquiring this data. * The fog layer (i.e., the middle layer or the bridge 
between the cloud layer and the latest layer) encompasses a large number of fog nodes, 
e.g., routers and gateways. They oversee scheduling, storing, data transmission and 
managing distributed computation. * The cloud layer is responsible for permanent data 
storage and extensive computational analysis of data. * The business layer is the data 
analytics and visualization layer which tends to user needs and specifications. 
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1. Repartition of the components and existing risks based on the common Fog computing 


architecture. 


We can encounter various issues at the diverse layers of the architecture, which in 


turn prevent IoT solutions to be fully operational. Issues are considered as faults, where 
faults are unintended defects that channelize to the cause of an error [8]. Hence, an error 
indicates the incorrect state of a system. Eventually, errors lead to the ineffectiveness of 
IoT systems. To describe a well-suited self-healing system, we construe the variety and 
nature of issues. Following IoT architecture depicted in Fig. 1, issues can be categorized 
into five major tracks as follows: 


1) 


2) 


Networking issues: The Edge layer consists in sensors, actuators, and gateways 
(Fig. 1). The networks that include these devices communicate with each other 
through a medium. In most cases, sensors/actuators communicate with the gateway 
through a local sensor network protocol (e.g., Bluetooth, Zigbee, Wibree, Zwave) to 
send the acquired environmental data/receive control signals. The gateway commu- 
nicates with the cloud nodes (i.e., cloud layer), through the fog layer, via the Internet 
supported protocols (e.g., LTE, 5G, Ethernet, Wi-Fi) to send the acquired data into 
the cloud database or receive control signals from the cloud. These communication 
links in the IoT system are frequently exposed to broken communication links, mal- 
functioning (e.g., incorrect state) or unavailability of the medium [4], which reduces 
the effectiveness of the overall system. 

Software issues: Software issues exist in all the layers (Fig. 1), including sensor 
reading, operating environment, faults in source code, services/application issues. 
* a) Sensors and/or actuators record environmental data and relay it to gateways. 
Hence, an error can occur at the reading moment leading to a sensor reading issues 
[8]. The sensor reading issue is an oscillation in the reading, which can be a flat 
signal giving an invariant repetition of random/arbitrary values. * b) The change in 
the operating environment [9] is an issue that affects badly gateways and cloud nodes. 
It is mainly related to a setting/configuration of a node that becomes not suitable 
upon an update. A real example of the change in the operating environment can 


3) 


4) 


5) 
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be mistaken in libraries, the direction of changes and version control [8]. Methods 
can be changed over version update and versions of components or libraries can be 
no longer compatible. * c) Faults in source code [8] which went unchecked in any 
prior patches often generate system/service/application failures. * d) Services and 
applications are also other kinds of the issue [10, 11]. The applications and services 
built upon the sensors can cause failures when they are in an incorrect state (e.g., 
on/off/crashed). 

Hardware issues: Hardware issues are one of the dominant aspects that entangle 
IoT systems regrouping: Energy depletion, Environmental conditions and Lack of 
memory and limited computational capacity. * a) Energy depletion is one of the best- 
known issues encountered by sensors. Indeed, once a sensor’s battery dies, it affects 
the application related to this sensor and its functionality. Also, it causes a major 
data loss [12]. Though considerable efforts have been carried toward fixing this issue 
by implementing an energy-aware, communication concept [4, 13] or expanding the 
battery capacity to sustain longer activity, the issue still remains. * b) Environmental 
conditions are also other aspects of the issue [14, 15]. Humidity, heat, vibrations or 
cosmic rays have a high impact on how sensors or gateways act [8]. Hereafter, these 
conditions can lower the sensor’s life or capacity preventing its optimal state. * c) 
Lack of memory and processors overheat are serious issues in IoT systems [4, 8, 9]. 
Since data is collected through sensors, a lack of memory resources leads to data loss 
and overload of the processors block the gateways or cloud servers from operating 
in their optimal way and even disable them. 

Human interaction issues: There is a high probability that human intervention can 
be part of a system failure [16] through: Implementation defects and operational 
mistakes, Co-programming and Formal Design. * a) Implementation defects and 
operational mistakes signify faults caused by human errors [8]. They are related to the 
process of deployment or nodes installation. * b) Co-programming is another form of 
human intervention issues [8]. This case involves a co-working environment where 
parts of code are uploaded without a proper upload process to avoid inconsistency 
and discontinuity. * c) Formal design enables building a common data model for the 
interaction with all the sensors despite their heterogeneity. Thus, a pattern that does 
not fit the sensors/actuators fields, prevent the gateways/cloud from processing the 
obtained data. 

Security issues: Security remains a major hurdle in IoT systems since datais growing 
fast and privacy is highly required. Security spreads through all the layers making 
it one of the important aspects to be taken care of [3, 17, 18]. From a data point 
of view, security flaws can be regarded from two perspectives: Data Security and 
Security of functioning. * a) Data Security regroups data stealing, data loss, and data 
privacy. The importance of self-awareness of issues compromising data has been 
documented [16]. Attackers are well versed in a data attack through techniques such 
as man-in-the-middle attack, phishing, intrusion to get the information they need. 
* b) The security of functioning restricts the nodes to work properly. Attackers 
can access nodes (i.e., Edge layer/cloud layer) through security techniques (e.g., 
phishing, jamming, flooding). Hence, they prevent the devices to work properly. 
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We present in the next section attempts that have been conducted to solve the 
aforementioned issues. 


3 Existing Self-healing Solutions 


Faults and failures are inevitable issues in computer-based systems, due to the inhos- 
pitable environment, unattended deployment, or unforeseen situations. Thus, researchers 
proposed numerous solutions to address these issues. Autonomic computing is the com- 
mon approach used in IoT to detect and resolve issues in an autonomous manner. This 
model regroups the capabilities of a self-healing system as follows; monitoring, ana- 
lyzing, planning, execution, and building knowledge. Following our review of IoT self- 
healing solutions: * Wallgren et al. [18] designed, implemented and evaluated a system 
called ‘SVELTE’. It is a lightweight intrusion detection system for the IoT. The work- 
flow of this system is as follows. First, it acts as a listener to gather the information 
around. Then, an intrusion detection component analyzes the mapped data and detects 
an intrusion. At last, a firewall was implemented to reduce the node overload by filtering 
unwanted traffic. * Dai et al. [19] proposed a system based on acquisition, detection, and 
reaction. It's a self-protected system based on feature recognition using virtual neurons. 
They proposed five self-protecting mechanisms each one of them equipped with an algo- 
rithm to sense, detect, and prevent possible attacks. * Mendonga et al. [3] proposed an 
architecture based on MAPE-K [20] loop, which consisted of a sensing phase present on 
all the nodes and a monitoring phase driven by an Artificial Neural Network Multi-layer 
perceptron (MLP). Information gathered by the latter phase was processed and analyzed 
to classify the information as safe, danger or inflammatory signals. Apart from secu- 
rity categories, hardware, software, and networking categories have also been studied. 
They include hardware failures, energy depletion or disrupted communications due to 
environmental conditions wind, rain, likewise communication link failure between the 
gateway and the sensors. * G. Gupta et al. [4] proposed an architecture for the runtime 
recovery of the sensors from the clusters in which the gateway has experienced faults. 
It's a detection and recovery architecture centered around communicating gateways. It 
can detect whenever one of the gateways suffers a fault and allocate the sensors to a new 
cluster where the gateway is functional. * Nguyen et al. [21] integrated the MAPE-K 
to a self-healing system for cyber-physical systems in smart buildings and cities. They 
emphasize building services for monitoring and processing data and the planning and 
execution of actions. * Angarita et al. [6, 22] proposed a self-healing system based on 
transactional web services. They worked on the recovery of web services upon failure, 
while Al-Dahoud et al. [15] emphasis on fault detection which is the first step to achiev- 
ing self-healing. * Another effective way to self-healing is IoT virtualization. It's easy 
to deal with services and software than hardware. Hence the capacity of a virtualized 
system becomes easily manageable. e.g., the concept of virtual objects has been invoked 
to overcome battery consumption and high-processing capabilities that lie in hardware 
and software categories [23, 24]. The virtual objects replace physical objects of IoT, 
consequently provide a lot of possibilities in terms of computation, storage and recovery 
over a fault or a failure [25]. Indeed, services can be handled more flexibly than hardware 
and can be monitored in different ways improving the capability of the system to recover 
from an unexpected state. 
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We present in the next section our AMI IoT case-based system and how the concept 
of self-healing of IoT is implemented. 


4 Self-healing Approach for AMI IoT Architecture 


The AMI-lab has been developing several IoT architectures for the past decade as infras- 
tructure to provide adaptable services for aging in place [7]. Based on our experience 
in developing and deploying IoT solutions, we decided to incorporate a self-healing 
solution that is not limited to security issues, but also targets hardware, software, human 
interaction, networking issues as well. 

The simplified presentation of AMI-IoT platform, as depicted in Fig. 2 includes four 
main layers (i.e., end-user environment, network, and cloud) which build the chain of 
communication. * End-user environment: includes all the environmental components 
such as sensors (e.g., physical nodes and sensors such as doors, motion, oxygen, bed- 
embedded sensor), actuators and nodes gathering the data. This component is the key 
point of the AMI-Platform, being the starting gathering point. * Fog Layer: includes the 
gateways which are tasked with the data modeling and preprocessing of data before it 
is transmitted to the cloud. * Network: includes all the components enabling the link 
between the end-user layer and the cloud layer and responsible for establishing secure 
communication. * Cloud layer: includes all the components enabling processes and 
store data (i.e., cloud nodes, database, computational nodes). This layer embeds the 
virtualization concept by replacing IoT physical objects with virtual objects. 

We adopted the autonomic computing approach, that is based on the three main 
phases (monitoring, analyzing, planning, execution, and knowledge), in implement- 
ing self-healing in AMI-Platform. Therefore, we hypothesized that the three elements, 
Sensing Approach (SenS), Awareness of Issues (AwaR) and Responsibility & Actions 
(ReAct), are prerequisites to address the IoT self-healing concerns in the three afore- 
mentioned layers. Additionally, the virtualization technique has been used and a set of 
self-healing agents/listeners are spread through the AMI-Platform (Fig. 3). 


4.1 Sensing Approach (SenS) 


The SenS element is the entry point of the architecture, and it is considered as the 
most important element. It ensures that the design and the goal of the architecture are 
both respected. If any unexpected event happens, it takes the responsibility to report the 
abnormality. The SenS can be regarded as the listening state of the AMI architecture. 
It is the foundation of the self-healing concept, enabling the power of monitoring and 
covering the behavior of the architecture. 

The AMI-Platform is composed of a set of components and the SenS is responsible 
for monitoring the whole chain from data detection (i.e.: by sensors/actuators) to its 
migration from environmental nodes (i.e., sensors to gateway) to the database (i.e., 
cloud nodes). To fully incorporate the SenS element, the cloud servers monitor all the 
components (i.e., from the end-user environment to the cloud architecture), but the 
components of the end-user environment should also monitor themselves. Knowledge 
is required to be gathered regarding the components of the AMI-Platform to be able to 


Self-healing Approach for IoT Architecture: AMI Platform 9 


Confirmation of 
the status of the 
system 


Level mitigation 
component 
Diagnosis 
Diagnosis - (AwaR) 
(AwaR) now 
violating event, è 
violated requiremé i ‘Seed 
involved syst 4 mitigation strate 


Updating the 
general knowledge 


Monitoring 
(Sens) 


Cluster-1 


Mitigation 
(ReAct) 


strategy 


Knowledge 


i Updating the low; 
H Monitoring level knowledge; 
i Sens] xS 1 " 
H — z PEL Mail alert 
4 Mitigation a 
DE (ReAct) $ 
` . ar oag id ái 
E Links — Low level process flow RR bow eve settheating components 
—+ High level process flow 
@ cluster head/gateways/Servers A " 
e dt = Shared resources |. | High level self-healing components. 
Cluster-n IŠ Healing actions [==> — Human interaction knowledge update 


Fig. 2. Self-Healing components in AMI-IoT architecture. 
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Fig. 3. AMI-IoT architecture with its device, fog and cloud layers. 


sense accurately and efficiently the components. This knowledge will serve as a mark 
for the working state of these components. Following SenS in the AMI architecture. 


e SenS in the end-user environment: It includes detection of battery level of all devices 
and sensors, data detection coming from nodes to the gateway, security level, network 
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stability, connection to the cloud nodes, the gateway’s processor state, and the services’ 
state. 

e SenS in the middle components: The SenS will be directed to the servers establishing 
the connection between the peers from the user environment to the cloud environment, 
the level of security, the services, and the working state(on/off). 

e SenS in the cloud environment includes the security level, the services, data retrieval, 
data insertion in the database, the working state of the database, the network, the 
storage, and the connection to the cloud nodes. 


4.20 Awareness of Issues (AwaR) 


The awareness element intends to make the system aware of abnormalities, which can 
occur and put the overall system in a disabled state. On this note, a system cannot 
be aware of failure state if the knowledge around the component is not accurate or 
well defined. Hence, though the SenS is important for gathering all the components 
information through monitoring, the AwaR plays its role by providing a parameter (1.e., 
what is an abnormality based on the sensed information and the knowledge). As a result, 
AwaR is related to the the layers of AMI-Platform as follows. 


e AwaR in the End-user environment: The IoT infrastructure in the end-user environ- 
ment is exposed to numerous issues preventing the data to achieve its main course. To 
face those challenges, AwaR relied on SenS to monitor the component and crosscheck 
the gathered information with the knowledge at hand. As an example, the battery level 
is one of the most prevailing issues in IoT. If the battery level falls below 5096, it means 
the battery is going to run out soon and there will be a discontinuity in the service. 
Consequently, a notification will be sent describing the sampling (i.e., data acquired 
by the sensors) as being too high or otherwise the encoding is too consumptive. * 
Services used by the gateway are another kind of issue. A service state is either on or 
off. Thus, an “off-state” defines the service as broken and impacts the transmission 
depending on how high the influence of the service is. Listeners are then applied to 
services that are vital to the system and upon a failure, the system notifies the issue 
stating the disabled state of the affected component. * Network state in one of the 
multiple issues invading the IoT architecture. The network might cut or appear as 
working but not working which results in the discontinuity of the data (i.e., Data stops 
being transferred from sensors to gateways or from gateways to cloud nodes). In this 
case, being aware means being able to notify that something is wrong with the connec- 
tivity across the AMI-Platform. On the security level, the firewall is deployed on the 
end-user side (i.e., gateways). Then, listeners to these firewalls have been developed 
to check the availability of the rules, a change in those rules, an intrusion into the sys- 
tem, communication with other peers, data anonymity and data leakage. Additionally, 
human interaction is strictly limited to avoid any change in the operating environment. 

e AwaR in Cloud architecture: it represents the core of the AMI-Platform. It's all the 
technologies and methods put together to enable a peer for each environmental node 
and the storing of the data in the database. Due to the knowledge accumulated and 
the SenS component, listeners applied to services will notify upon failure and point 
out the exact service/component which is having an issue. Network issues are rare at 
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this point and battery issue is nearly nonexistent. On the security level, listeners are 
developed and applied to the firewall to notify at the slightest change in the firewall 
table and an intrusion that occurred. 

Regarding the security of the data in the database, everyone who is not allowed to 
access the data will be reported as soon as an attempt is made. Since the peer (i.e., cloud 
nodes getting the data from the gateway in the end-user environment) is virtualized, 
when the peer system encounters a high-level failure, the system should also notify 
the situation in an optimal amount of time. Another critical point is the storage of 
the data. AMI-Platform deployed listeners to notify when the storage capacity on the 
cloud servers reached a certain point so that servers will not be overloaded before 
actions are taken. 

Regarding human interaction, a pipeline has been deployed to notify all the 
programmers when a code has been uploaded. Validation of the code is then pro- 
cessed to check the continuity with the overall code. In this way, there will be fewer 
co-programming issues and likewise fewer failures upon deployment. 

e AwaR in Network: Named in IoT architecture, the weak link, due to its public nature, 
it can be subject to many issues mentioned in the previous subsection. Listeners have 
been improved and applied to the security level for the firewall table. Notifications will 
be sent upon addition, subtraction or change in the table and for intrusion detection 
in the system. The system will notify as well when there is a missing link between 
the end-user node and the peer in the cloud. Services that are monitored by the SenS 
will be applied listeners as well to make the system aware of the availability of the 
services. 


4.3 Responsibility and Actions (ReAct) 


The ReAct element confers the "responsibility" feature to the AMI-Platform, which 
enables the autonomy in decision-making (Angarita and Kelaidonis et al. work [6, 23, 
24]). A responsible component is achievable only because the SenS and AwaR elements 
exist. The more elaborated knowledge, the more accurate data and the more efficient the 
method to deal with an unexpected event. Hence, making a system able to take actions, 
depending on the outcome of a situation is the key role of this part. Toward a meticulous 
Work, we strive to achieve a self-healing architecture. Since "responsibility" is the way, 
a component autonomously manages itself, we target to spread the concept through the 
all the layers of AMI-Platform. Following concrete examples of ReAct implementation 
in AMI-Platform. 


e ReAct in the End-user environment Concerning the battery level, actions will be 
taken dynamically on the sampling ratio or the encoding format of the data. There- 
fore, the battery consumption will drop, and the battery life will be maintained for 
a longer period. As for services that are monitored, they can be restarted upon fail- 
ure. Regarding the network state, the link can be reestablished after an idle time. 
Afterward, notification is sent upon resolution. As for security, AMI-lab deployed a 
set of agents responsible for reconfiguring the firewall table upon a change detection 
and logging out a user upon an intrusion detection and reconfiguring the whole gate- 
way environment to ensure the same level of security is respected. Other agents are 
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deployed for data detection and transmission and the communication with the other 


peer in the cloud. 

e ReAct in Cloud architecture for this part, agents are used in the same way regard- 
ing the security level, acting upon change detection in the firewall table, intrusion 
detection, service failure. Regarding the storage management, agents that have been 
deployed will clean up space and make more space than notify about what happened 
and which course of action has been taken to deal with the ongoing issue. 

e ReAct in Network Agents have been deployed much more regarding the security 
level. 


5 Testbed Implementation 


To evaluate the reliability of the proposed model, in this section we first focus on the 
implementation of each component used, located at different layers of the system, then 
we present and discuss the obtained results. 


5.1 System Architecture 


A bed-embedded sensor, a motion sensor and a smart IoT gateway were used in the 
IoT end-user layer, whereas and an IoT virtual gateway and servers were used in the 
IoT Cloud layer. Bed-embedded sensor: The sensor used in our system, monitors and 
collect ambient temperature, vital features such as heart rate, wake-up time, sleep time, 
total time of sleep, bed interruptions and the out of bedtime. The sensor used is a fiber 
optic mattress. Motion sensor: The sensor used in our system, is an event sensor reacting 
to movement in a specific area, collecting upon detection. 

Both sensors send the collected information to an IoT smart gateway which will 
process the information. The motion sensor is connected to the gateway through Zwave 
protocol which is a low power wireless protocol. However, the bed-embedded sensor 
sends information through a serial connection which is physical and wired. 


Smart IoT Gateway: We used a Raspberry Pi 3 model B with a 1.2 GHz Quad-Core 
ARM Cortex processor, 1 GB of RAM, and which is permanently connected to an 
electrical power supply and located in the person room. To allow interoperability of the 
heterogeneous sensors, Raspberry Pi is equipped with several communication modules. 
Therefore, a Zwave controller was used to establish a communication with the motion 
sensor, while a serial connection was used to interact with the bed-embedded sensor. The 
overall information is then processed and sent over MQTT protocol. An MQTT broker 
has been implemented on the Raspberry Pi to enhance light communication between 
two entities i.e., one publishing and the other one subscribing. 


IoT Virtual Gateway: It’s a virtual smart gateway based on python programming lan- 
guage and virtualization technology. It leverages virtualization and through an MQTT 
client (paho-mqtt) subscribes to the information published by the MQTT broker. The 
received information is processed and then stored inside a database (Elasticsearch). 


Servers: The servers are built through a virtualization platform (Vsphere). Regarding 
the database, we used Elasticsearch coupled with Kibana to enable full-text search and 
process the information in different ways. 
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5.2 Results and Analysis 


The experimentation has been conducted over 4 days. We compared the results with 
an architecture without self-healing components and an architecture with self-healing 
components. Upon analysis, several components revealed to be important for the purpose 
of the architecture. They are mainly: 


Internet connectivity: It ensures data is not lost and sent properly and keeps track of 
all the downtimes and uptimes. 

Connectivity between the gateway and the virtual gateway: Ensures the connection 
between the physical nodes and the virtual nodes in the cloud to have a reliable data 
communication channel. 

Services: They are responsible for collecting the data, modeling the data and pre- 
processing it before publishing it through MQTT. 

CPU load: Ensures the gateways are not overloaded. 

e Storage: The Kubernetes system produces a lot of data which can, once reached a 
threshold, hinder the proper working of the system. 


On a prediction base, the amount of data which is supposed to be gathered by the 
end of the 4" day is the Awaited Data (AWD). The acquired data will be titled as ACD, 
the number of actions that have been taken and fixed the issue will be titled NAS, while 
the number of actions that failed is titled NAF. Other metrics such as the Data Coverage 
(DC), Failed Self-Healing Rate (FSHR), the Successful Self-Healing Rate (SSHR) and 
the Self-Healing Model Accuracy (SHMA) are described to measure the performance 
of our system. 


Data coverage (DC): Is the Acquired Data (ACD) over the Awaited Data (AWD). This 
metric is used to verify the successful data acquired within a timeframe. 

Failed Self-Healing Rate (FSHR): It reflects the failed actions (NAF) over the total of 
actions taken (NTA). It represents the misunderstanding of the system. 

Successful Self-Healing Rate (SSHR): Itis the successful actions (NAS) over the total 
number of actions (NTA). It represents the understanding of the system and how much 
of the actions have been taken to fix an issue. 

Self-Healing Model Accuracy (SHMA): It is the measure of the overall Self-Healing 
component which determines the accuracy of the Self-Healing component design. 
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Table 1. Metrics before self-healing. Table 2. Metrics after self-healing. 
Métrics Hits Métrics Hits 
ACD 73920 ACD 1331057 
AWD 1774080 AWD 1774080 
DC 0,04(4%) DC 0,75(75%) 
NAF 1 
NTA 92 
FSHR 0,010(196) 
NAS 91 
NTA 92 
SSHR 0,98(98%) 
SHMA 0,99(99%) 


Finally, over the four days, several values were recorded. Looking at the system 
without the self-healing components, the awaited data was 1774080 hits whereas the 
acquired data was 73920, giving a DC of 4,2%. Compared with the system coupled with 
the Self-Healing model the values are different. They were for the awaited data 1774080 
hits, the acquired data 1331057 producing a DC of 75%. From those result, we noticed 
that the implementation of a Self-Healing model, makes the system reliable. The results 
can be seen through Tables | and 2. 


6 Discussion 


Apart from the solutions described in Sect. 3, we argue that the self-healing in IoT can 
benefit from the virtualization and the test coverage technics. To start with, Applica- 
tions/Firmwares that are developed in a way to incorporate unit tests and log functions 
(i.e., test coverage) make the system alive and make the sensing phase more complete. 
Additionally, developers that feed IoT applications with the possible issues that can 
arise and the way to solve them, from the start (at the development and deployment 
phases), re-enforce the self-healing concept. Hence, the system is already prepared to 
sense (SenS) the environment, detect the identified issue (AwaR) and react accordingly 
(ReAct). Furthermore, from the real-world applications of IoT experiments, information 
and issues shall be gathered more to make an efficient aware system. Consequently, the 
more knowledge gathered the more efficient the self-healing system. 

Bringing forth the virtualization concept in the process, the AMI-Platform empha- 
sized replacing the physical objects by a virtual object. The platform handles more 
flexibly the process of recovery and embeds the concept of microservices. Thus appli- 
cations/services are made modular to enable significant adaptability and scalability of 
the platform offering numerous possibilities Kelaidonis et al. [26]. 
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7 Conclusion 


Failures are bound to happen in today IoT solutions. Thus, we focus in this paper on 
the approach to address these issues. We start in this paper by reviewing the progress 
of the research works related to IoT issues and classify them into five categories (i.e., 
Networking, Software, Hardware, Human interactions and Security issues). Moreover, 
to mitigate such issues, proposed solutions from researchers are analyzed. We found that 
most of the efforts focus only on one issue, none focuses on a comprehensive approach 
that addresses all the five. 

We reflected in this paper on our approach that covers most of the issues. It used 
the concept of self-healing agents that act in listener, detector and healer modes. The 
listener mode allows the agents to sense the environment in the main layers of IoT 
architecture (i.e., Device, Fog and Cloud layers). This mode enables the collection of 
data from vital components (e.g., network, security, services). Through the knowledge 
and the sensed information, the detector mode crosschecks the information to get an 
accurate detection of an abnormality in the system (i.e., network state down, services 
down, security lacking). Finally, the healer mode defines the proper course of action to 
take to revert the system to a normal state. 

Moreover, instead of a centralized monitoring/self-healing system, we opted for a 
agent-based distributed self-healing system which focus on spreading the agents through 
allthe layers. This system builds an optimal and autonomous way to recover from a failure 
right after an unexpected situation occurs. 

We also discussed in this paper the prerequisites (1.e., knowledge, listeners, detectors, 
healers, virtualization and test coverage technics) to deploy a scalable and resilient IoT 
infrastructure in order to ensure a self-healing mode. We target in our future work a 
more efficient IoT system with Quality of Service (QoS) integration via developing an 
advanced monitoring approaches and fault tolerating models which are to be based on 
data-analytics and reasoning algorithms over the large amount of data collected. 


References 


1. Rivera, J., Goasduff, L.: Gartner says a thirty-fold increase in internet-connected physical 
devices by 2020 will significantly alter how the supply chain operates. Gartner (2014). 
https://www.gartner.com/en/newsroom/press-releases/2014-03-24-gartner-says-a-thirty- 
fold-increase-in-internet-connected-physical-devices-by-2020-will-significantly-alter-how- 
the-supply-chain-operates. Accessed 8 Sep 2021 

2. Naik, S., Maral, V.: Cyber security — IoT. In: 2nd IEEE International Conference on Recent 
Trends in Electronics, Information & Communication Technology (RTEICT) (2017). https:// 
doi.org/10.1109/RTEICT.2017.8256700 

3. de Almeida, F.M., Ribeiro, A., Moreno, E.D.: An architecture for self-healing in internet of 
things. In: Ubicomm (2015) 

4. Gupta, G., Younis, M.: Fault-tolerant clustering of wireless sensor networks. IEEE Wirel. 
Commun. Networking Conf. WCNC 3, 1579-1584 (2003) 

5. Vecchio, M., Sasidharan, S., Marcelloni, F., Giaffreda, R.: Reconfiguration of environmental 
data compression parameters through cognitive IoT technologies (2013). https://doi.org/10. 
1109/WiMOB.2013.6673353 


16 


23. 


24. 


25. 


26. 


B. Abdulrazak et al. 


Angarita, R.: Responsible objects: towards self-healing internet of things applications (2015). 
https://doi.org/10.1109/ICAC.2015.60 


. Abdulrazak, B., Paul, S., Maraoui, S., Rezaei, A., Xiao, T.: IoT Architecture with Plug and 


Play for fast deployment and system reliability: AMI Platform. In: International Conference 
On Smart Living and Public Health (2022) 

Raghunath, K.K.: Investigation of faults, errors and failures in wireless sensor network: a 
systematical survey. Int. J. Adv. Comput. Res. 3(12), 3 (2013) 

Koopman, P.: Elements of the Self-Healing System Problem Space. System (2003) 

Joshi, K.R., Hiltunen, M.A., Sanders, W.H., Schlichting, R.D.: Automatic model-driven 
recovery in distributed systems (2005) https://doi.org/10.1109/RELDIS.2005.11 


. Porter, B., Ta`iani, T., Coulson, G.: Generalised repair for overlay networks (2006) 
. Lazarescu, M.T.: Design of a WSN platform for long-term environmental monitoring for IoT 


applications. IEEE J. Emerg. Sel. Top. Circuits Syst. 3(1), 45-54 (2013) 


. Singh, S., Woo, M., Raghavendra, C.S.: Power-aware routing in mobile ad hoc networks 


(1998). https://doi.org/10.1145/288235.288286 


. Begum, R., Shaikh, Syed, A.: Sensor node failure detection in wireless sensor network: a 


survey (2018) 


. al Dahoud, A., Doghmane, R.: Failure detection on wireless sensor network based on 


comparative study (2017). https://doi.org/10.1109/RISE.2017.8378140 


. Sadek, L, Rehman, S.U., Codjo, J., Abdulrazak, B.: Privacy and security of IoT based health- 


care systems: concerns, solutions, and recommendations. In: Pagán, J., Mokhtari, M., Aloulou, 
H., Abdulrazak, B., Cabrera, M.F. (eds.) ICOST 2019. LNCS, vol. 11862, pp. 3-17. Springer, 
Cham (2019). https://doi.org/10.1007/978-3-030-32785-9 1 


. Tari, Z.: Security and privacy in cloud computing. IEEE Cloud Comput. (2014). https://doi. 


org/10.1109/MCC.2014.20 


. Raza, S., Wallgren, L., Voigt, T.: SVELTE: real-time intrusion detection in the internet of 


things. Ad Hoc Netw. (2013). https://doi.org/10.1016/j.adhoc.2013.04.014 


. Dai, Y.S., Hinchey, M., Qi, M., Zou, X.: Autonomie security and self-protection based on 


feature-recognition with virtual neurons (2006). https://doi.org/10.1109/DASC.2006.24 
Kephart, J.O.: Research challenges of autonomic computing (2005). https://doi.org/10.1145/ 
1062455.1062464 


. Nguyen, T.A., Aiello, M., Yonezawa, T., Tei, K.: A self-healing framework for online sensor 


data (2015). https://doi.org/10.1109/ICAC.2015.61 


. Angarita, R., Rukoz, M., Cardinale, Y.: Modeling dynamic recovery strategy for composite 


web services execution. World Wide Web 19(1), 89-109 (2015) 

Foteinos, V., Kelaidonis, D., Poulios, G., Vlacheas, P., Stavroulaki, V., Demestichas, P.: Cogni- 
tive management for the internet of things: a framework for enabling autonomous applications. 
IEEE Veh. Technol. Mag. (2013). https://doi.org/10.1109/MVT.2013.2281657 

Kelaidonis, D., et al.: Virtualization and cognitive management of real world objects in the 
internet of things (2012). https://doi.org/10.1109/GreenCom.2012.37 

Atzori, L., etal.: SDN&NFV contribution to IoT objects virtualization. Comput. Netw. (2019). 
https://doi.org/10.1016/j.comnet.2018.11.030 

Kelaidonis, D., et al.: A cognitive management framework for smart objects and applications 
in the internet of things. In: Timm-Giel, A., Strassner, J., Agüero, R., Sargento, S., Pentikousis, 
K. (eds.) MONAMI 2012. LNICSSITE, vol. 58, pp. 196-206. Springer, Heidelberg (2013). 
https://doi.org/10.1007/978-3-642-37935-2 15 


Self-healing Approach for IoT Architecture: AMI Platform 17 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter's Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter's Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Check for 
updates 


Digital Twin Driven Smart Home: 
A Feasibility Study 


Alireza Asvadi®), Andrei Mitriakov, Christophe Lohr, 
and Panagiotis Papadakis 


IMT Atlantique, Lab-STICC, UMR CNRS 6285, Team RAMBO, 29238 Brest, France 
{alireza .asvadi,andrei.mitriakov,christophe.lohr, 
panagiotis.papadakis}@imt-atlantique.fr 


Abstract. We aim to facilitate the daily-life activities of frail or elderly 
people in collaboration with mobile assistive robots through the means 
of a digital twin-powered smart home. Being able to quickly and effi- 
ciently produce a digital twin of the human user’s environment, can help 
to further develop personalized assistive solutions. As our first investi- 
gation toward this goal, we describe our proof-of-concept “digital twin- 
driven smart home” implementation. It consists of a virtual represen- 
tation, robot navigation and environment semantics using open-source 
software. The initial obtained results on the building process of the dig- 
ital twin are encouraging and suggest the possibility of integration of 
digital twin for smart spaces. 


Keywords: Ambient assisted living - Cyber physical system - 
Living-lab - Semantics 


1 Introduction 


In recent years, the availability of low-cost sensors and open-source middle-ware 
software have opened up interesting research areas in the robotics field. In partic- 
ular, the next generation of simulation models called Digital Twin (D'T), which 
represents a continuous virtual replica of a physical system, has gained increas- 
ing attention. It can be used to create a simulation of a smart home including 
assistive/service robots and human users. It has applications in the optimization 
of robots and smart home settings. For example, finding the optimal number and 
configuration of sensors especially when new robots or users are introduced. As 
another examples, monitoring in real-time, further analysis and learning of edge 
cases and rare situations in DT for the safety of human users, e.g., by pushing 
those events from simulation to real-world and vice-versa (thanks to DT that 
enables a bi-directional link between simulation and real-world). In practice, 
such optimization should be carried out over a variety of houses and users. 

In this context, this paper aims to put forward the use of DT technology in 
the development of a smart home system. The main contributions are: 


e This work looks at the practical aspects of the building process of DT. 
e Provides analysis and insights into the difference in the performance between 
the actual and simulated environments. 
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The paper is organized as follows. Section 2 gives a definition and recent related 
works. Section 3 details the proposed DT development process. Experimental 
results are described in Sect. 4, and Sect. 5 brings some concluding remarks and 
proposes directions for further research. 


2 Background 


With the progress of the internet of things and artificial intelligence, several 
strategies were developed to advance smart technologies, including Industry 4.0 
[11] and Society 5.0 [4]. At the core, these technologies are based on a cyber- 
physical system (CPS) that integrates physical and digital entities. In this con- 
text, DT is an approach to implement cyber-physical integration. The DT is a 
digital representation of a physical system that precisely models and can predict 
how the actual system will perform. In this respect, it serves as a tool to bridge 
the gap between physical and digital worlds. 


Definition. Some of the commonly accepted concepts in the definition of the 
DT are the following: it is a digital representation of a physical entity (including 
geometric, functional, and usage models), being ultra-realistic and fully consis- 
tent with the physical entity, continuous and automatic update to be able to 
adapt to changes, semantic content to support intelligent decision making, and 
implementation in a decentralized structure to collaborate with other DTs [17]. 


DT for Robotics. While there is a rich literature on DT concept [5,8], there 
have been few reports on the 3D modeling aspect in robotics and smart home 
applications. Kousi et al. [9] combined multi-sensor data and CAD models to 
create a virtual representation (including semantic and dynamic update compo- 
nents) of a production environment. Havard et al. [7] proposed a co-simulation 
approach between DT and virtual reality in a human-robot collaborative work- 
place. Another notable work is Habitat 2.0 [16] which is a platform for train- 
ing simulated robots in interactive environments. Phanden et al. [13] recently 
reviewed existing simulation software for DT development. For robotics, they 
found Gazebo and V-REP [15] for modeling sensors and image-based systems, 
ROBOTRAN to model multi-body systems, and ROBCAD for multi-device 
robotic and automated manufacturing processes. 


DT for Smart Home. As to digital models of buildings, To et al. [18] per- 
formed drone-based reconstruction for D'T augmentation of buildings. Calderita 
et al. [1] proposed a CPS for ambient assisted living including humans and assis- 
tive robots in a care-giving center. A related term to DT is Building Information 
Modeling (BIM). Czerniawski and Leite [2] reviewed approaches for BIM. They 
identified the following steps for model creation: collecting sensor data and 3D 
reconstruction, semantic and geometric modeling for the recognition of semantic 
classes in sensor data and describing the instances! shapes, and finally the cre- 
ation of BIM that integrates semantic and geometric components. In comparison 
with BIM (which is one-way modeling), in a DT there is a bi-directional data 
flow between the virtual and physical entities. 
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Fig. 1. The conceptual digital twin of the smart home. The dashed-red arrows mark 
the process that are not implemented in a fully automated way. The white box shows 
the component that is developed primarily for a feasibility study. (Color figure online) 


'There are a number of other works and projects that discuss related concepts 
on ambient assisted living and the synergy of ambient intelligence with robots. 
Examples are GIRAFF-+ that consists of a network of smart home sensors and 
a telepresence robot to monitor elderly people, and STRANDS that explores 
adaptation to changes over time for mobile robots in 3D dynamic human envi- 
ronments. Previous work of the authors concentrated on the potential services 
that could be offered from such systems [12]. 


3 DT-Powered Smart Home 


The DT of a smart home is defined as: using a priori information (e.g., archi- 
tectural plans, input from inhabitants) and given input sensory data (e.g., home 
automation and robot sensors), build a continuous model including robots and 
user models. As a first step toward this goal, a prototype DT model of the smart 
home has been developed. Figure 1 demonstrates an abstraction of our under- 
standing of what a DT workflow should consist of. Ideally, such a workflow has 
limited dependence on human supervision so as to reduce errors and favor porta- 
bility and generalization. The steps are defined as follows. The smart home is 
scanned using a hand-held RGB-D camera and a SLAM algorithm to build a 
reconstruction of it. A modeling software is used to construct the CAD models 
of the home and objects inside it from the scan. The CAD models and a robot 
model are imported into a robotic simulator. A proof-of-concept is developed to 
enable semantic understanding of objects. 

The DT can be conceptualized in data, algorithmic and model. The data 
layer represents the communicated data including point cloud, navigation (veloc- 
ity and pose), etc. The algorithmic layer includes software to develop the mod- 
ules. The model layer comprises the geometric, functional and semantics includ- 
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ing the CAD models, a TurtleBot!, and a pre-trained object detection model. 
More information can be found on the project website?. 


3.1 3D Reconstruction 


To have a metric measurement of the actual environment a hand-held scanning 
is performed. The Orbbec Astra Pro RGB-D camera is used which is shown to be 
one of the best low-cost sensors for the reconstruction of indoor spaces [3]. The 
RTAB-Map (Real-Time Appearance-Based Mapping) [10] algorithm is used. It 
consists of loop closure and proximity detection (using a bag of words), graph 
optimization (to decrease odometry drift) and global map assembling (using local 
occupancy grid). It requires an odometry estimation, i.e., motion estimation 
between consecutive scans. This is performed by computing visual odometry 
using RTAB-Map and the camera. The process outputs a dense point cloud. 
The CAD models are created from point cloud data. A modeling software is 
used to craft the CAD model of the home structure (e.g., walls) from the point 
cloud, also online public repositories are used to obtain the models of the real 
home items. It should be noted that an automatic transformation of 3D scans 
into CAD is an open question (see e.g., Scan2CAD project) that will not be 
addressed in our work. 


3.2 Physics-Based Simulation 


The Gazebo is used to simulate the dynamics of the system. A digital 3D model 
is developed to represent the smart home environment and the objects in the 
simulator. The CAD models are used to define the visual and collision properties 
of the items in this model. Other physical properties can be defined, for example 
pose, mass, being static (e.g., walls) and dynamic (e.g., objects). A TurtleBot 
model is spawned in the simulated system. The use of Gazebo with ROS enables 
communication between the simulated and the real system. 


3.3 Semantics 


The detection of the categories of objects of interest (e.g., tables and chairs) and 
their position can have several benefits in a DT framework. The You Only Look 
Once (YOLO)-v3 algorithm [14] is used for object detection. It reads camera 
image data and returns the 2D bounding boxes. The pre-trained model is able 
to detect 80 classes including chair, sofa, table, bottle, microwave, person, etc. 
The darknet_ros_3d package? is used to add the 3D bounding boxes of objects to 
the YOLO. It combines the detected 2D bounding boxes with point cloud data 
to calculate the 3D bounding boxes. 


1 https:/ /www.turtlebot.com/turtlebot2/. 
? https://sites.google.com/view/heron-project. 
3 https://github.com/Intelligent RoboticsLabs/gb. visual. detection. 3d. 


22 A. Asvadi et al. 


Fig. 2. Top-left: Top panoramic view of the smart home. top-right: Illustration of an 
area in the reconstructed map (from the view shown by a red arrow in the left image). 
Bottom: the full reconstructed map (from top view). 


4 Results and Discussion 


Sections 4.1 and 4.2 describe the process results for the creation of the digital 3D 
model and a comparison of the DT and the actual system. Section 4.3 provides 
discussion on the semantics aspect. The approach is evaluated in a small exper- 
imental flat including a kitchen, a bathroom and a living room with furniture 
(see Fig. 2). 


4.1 Creation of the Digital 3D Model 


3D Reconstruction. A detailed colored 3D point cloud of the flat is achieved 
which has about 700k points (see Fig. 2). The following are some of the observed 
limitations. 


e The windows need to be covered in sunlight. The Astra Pro camera works 
based on infrared technologies. The light coming from the infrared projector 
can be outshined by the sunlight and no point cloud can be computed. 

e The mirror, window glass, TV surface, and black colored objects degrade the 
quality of the depth image and the point cloud. 
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e The textureless walls, ground and ceiling must be avoided or rectified (e.g., by 
adding picture frames to plain walls). Because the visual odometry in RTAB- 
Map is based on extracted features from the RGB-D images. Therefore, in 
environments without enough features, the odometry cannot be computed. 


Digital 3D Model. An idealistic 3D model of the flat was made by a team 
of students. The model was built by first measuring different dimensions of the 
flat and then recreating it using a modeling software. A quantitative evaluation 
is performed with a comparison of our CAD model against this idealistic model 
(see Fig. 3-Top). To perform the evaluation the models are converted to point 
cloud data using ray tracing operations. The root mean squared error (RMSE) 
and Hausdorff distance metrics (which are common general-purpose metrics for 
comparing 3D shapes) are used to estimate the differences (see Table 1). The 
accuracy of about 9cm average RMSE is obtained which is encouraging and 
demonstrates an interesting perspective for an automated reconstruction-based 
model creation. As it can be seen in Fig. 3- Top, the perpendicularity of the walls 
in our developed model (shown in red) are preserved well with only a slight 
slope on the right side. Figure 3-Bottom shows the furnished smart home with a 
simulated TurtleBot in the Gazebo. 


Table 1. A comparison of the quality of the developed smart home’s CAD model 
against the idealistic model. 


Metrics RMSE | Hausdorff 
Error (in m) |0.088 | 1.354 


4.2 Robot Navigation 


Experiments were performed to compare the traversed trajectory and the gen- 
erated 2D maps in the actual and simulated conditions. 


Trajectory Evaluation. A velocity message with the linear and angular values 
of +0.25 and —0.1 is published (with the frequency 10 Hz for 4s) to control the 
translational and rotational speed of the robot. The objective is to perform the 
navigation in the D'T and reality, and to compare the trajectories. 

To obtain the trajectory of the actual Turtlebot, a visual marker was used. 
We position the marker on the upper part of the Turtlebot in the field of view 
of a high-resolution camera (see Fig.4). The robot poses can be retrieved by 
detecting the marker in the camera image. A toolkit tracks marker poses which 
enables tracing of the Turtlebot trajectory in the camera coordinate frame. The 
experiments were performed in 5 iterations. The average and standard deviation 
of the trajectories in both real and simulated cases were reported in Fig.5. In 
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Fig. 3. Top: Illustration of the representations. Superimposed simplified mesh 
(obtained from the point cloud), the model created from point cloud (red), and the 
idealistic model (green) for a visual inspection. Bottom: The simulated smart home 
with a Turtlebot model. (Color figure online) 


addition a simplified analytical solution for the expected trajectory of a robot 
performing uniform circular motion is computed. It is equal to traverse of 0.4 rad 
on the perimeter of a circle with radius of 2.5 m. It should be mentioned that the 
complete Turtlebot model is a differential drive consisting of two wheels and two 
caster wheels which was not considered here. It was noticed that the turtlebot 
in the real condition traversed a shorter distance in the trajectory (about 25 cm 
difference with the simulated one). This was probably because of the uncertain- 
ties on the friction parameters between the robot and the ground surface. Due to 
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Fig. 4. Left: The trajectory evaluation experiment. Right: Left and right show the 
Turtlebot at the start and end of the experiment. Top and bottom show the simulated 
and actual setup. 
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Fig. 5. The average and standard deviation of the obtained trajectories in the real and 
simulated situations (the starting point is on the left-side of the curves). 


Fig. 6. Illustration of the 2D occupancy grid maps. From left to right: the generated 
grid map in the simulated environment, the created map in the actual home, the 
obtained map after the additional rotation in the real case. 
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Fig. 7. 2D Object detection using YOLO-v3. The images show the detection results in 
different areas in the smart home. 


using a simplified model, there were also differences in the recorded trajectories 
in comparison with the analytical solution. 

Understanding the difference between the virtual model and the real world 
and adjusting such parameters can improve the robot’s behavior in reality. It 
helps to accurately estimate and plan the robot trajectory and in the application 
level to provide both reliable and consistent service for elderly users in the smart 
home. 


Mapping Experiment. A 2D mapping process is performed and compared in 
the DT and reality. The turtlebot rotates 360° (with the angular speed of 10° per 
second) to cover the whole space. The gmapping* package from ROS is used [6]. 
It is a laser-based SLAM. To be able to work with a RGB-D camera, it converts 
a depth image to a fake laser scan. It uses particle filters (each particle carries 
an individual map) and several adaptive techniques to learn the occupancy grid 
map of the environment. Figure 6 shows examples of the generated maps. It was 
observed that the actual turtlebot could not cover the full profile of the home 
using the rotation message of 360°. The rotation of 540° was published to the 
actual turtlebot to fulfill the coverage of the whole area. 


4.3 Semantics 


Object detection can be used as a coupling mechanism of the real and the digital 
worlds. As a feasibility study, 2D and 3D versions of YOLO object detection were 
employed to detect the objects in the smart home. 


* http:/ /wiki.ros.org/gmapping. 
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Fig. 8. 3D Object detection using YOLO-v3 and Darknet ROS 3D. Left: shows a person 


standing and moving in the smart home. Right: shows results in another setting with 
different furniture. 


Quantitative Analysis of 2D Object Detection. Figure 7 shows 2D object 
detection results in different areas of the smart home. An evaluation is performed 
to quantitatively measure the object detection performance in the smart home. A 
number of 30 detection frames were selected randomly and the precision, recall, 
and F1 scores computed for the two most frequent items seen in the smart home: 
the chairs and TV monitors, with the total number of 50 chairs and 41 monitors 
observed in all images. As it can be seen in the Table 2, the detection rates were 
different for each item. For example, the TV monitors were detected with higher 
precision whereas the detector has a better recall value on chairs. 


Table 2. A comparison of 2D object detection in the smart home. 


Object/Metrics | Precision | Recall | F-Score 
Chair 0.53 0.88 0.66 
Monitor 0.97 0.68 | 0.80 


Qualitative Analysis of 3D Object Detection. Figure8 shows an example 
of the 3D detection results. The top shows the 2D detections. The yellow boxes 
in the bottom show projected detections into the point cloud in the 3D space. 


5 Conclusions 


In this paper, we presented an approach for the generation of a digital 3D model 
of asmart home using open-source software and a low-cost sensor. The methodol- 
ogy and initial results (including the obtained models) are promising and appear 
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to have a potential for the design of DT-driven smart homes. However, further 
research is needed to explore the integration of the digital profiles of the con- 
nected home automation sensors (e.g., thermostat and pressure sensors) into 
the DT. The next challenge is to extend the developed DT to include other 
entities: sophisticated robots (than a simple TurtleBot) and human users (e.g., 
to sense where the user is and what is doing). Another interesting directions 
are benchmarking, further analysing and quantifying the different steps of DT 
creation. 
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Abstract. Formal method techniques are used to model complex sys- 
tems as mathematical entities. By building mathematical rigorous mod- 
els of IoT design patterns, it is possible to verify their properties in a 
thorough fashion. In this paper, we propose a refinement-based approach 
for modeling IoT design patterns. It allows the modeling of correct by 
construction IoT design patterns. It takes advantage of formal methods 
by the specification of design pattern models with the Event-B method 
and checking the design correctness. Our goal is to design IoT patterns 
proven correct by construction to successfully apply them and promote 
their reuse. Our approach is experimented through pattern examples and 
we illustrate it with a case study in the health care domain. 


Keywords: IoT design patterns - Software architecture modeling - 
Pattern modeling - Formal specification 


1 Introduction 


System architectures are becoming more complex in their understanding and 
analysis. Among these architectures, we find the architecture of Internet of 
Things (IoT). The IoT is a recent concept. It is a variety of smart objects 
interconnected via the internet. This variety of objects faces the problem of 
complexity of the system. This complexity opens the way to the use of a variety 
of IoT design patterns to address many issues [10]. They propose solutions for 
common and recurring problems to architects and designers in the IoT domain. 
Most of these patterns are presented visually and informally, there is no formal 
semantics associated with them. Hence, their meanings may be imprecise. They 
can lead to their misunderstanding and use. 

Faced by this situation, and to remedy the problem, we propose an app- 
roach that allows to model and specify these patterns with a formal notation 
that allows to reuse them correctly. Our objective is to prove the relevance of 
these patterns. We propose an approach, allowing the formal modeling of these 
patterns in order to describe both their structural and behavioral features. Struc- 
tural features are generally specified by the types of entities. The configuration 
of the entities is also described in terms of static relationships between them. 


© The Author(s) 2022 
H. Aloulou et al. (Eds.): ICOST 2022, LNCS 13287, pp. 30-42, 2022. 
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Behavioral features describe successive interactions between the different enti- 
ties of the IoT application. The proposed formal specification of IoT patterns is 
generic and it is based on the formal Event-B method. We apply our approach 
on different pattern examples and we illustrate it with a case study in the health 
care domain. 

The rest of this paper is organized as follows. Section2 discusses related 
works. Section 3 describes how to formally specify IoT design patterns with the 
Event-B method. In Sect.4, we present an application to a case study of our 
approach. Section 5 concludes and gives future work directions. 


2 Literature Review 


Most of the proposed design patterns (Object-Oriented design patterns, Enter- 
prise Application Integration (EAI) design patterns, Service-Oriented (SOA) 
design patterns, and Connected Object (IoT) design patterns) are described by 
a combination of a text description and a graphical representation sometimes 
using a proprietary notation in the aim of making them easy to understand 
like SOA design patterns [5] and Object Oriented design patterns [6]. However, 
these descriptions make patterns ambiguous and may lack details. Some work 
so have proposed the semi-formal representations of these patterns using mod- 
eling languages [4]. Some other works use or provide formal languages based on 
mathematical notation for a precise pattern specification [16]. However, these 
approaches require knowledge of mathematics and first order logic to use them. 
Some research has chosen to combine the semi-formal and formal representations 
of patterns. This representation ensures a better understanding and precision of 
patterns. Generally speaking, there is a consensus on the elements that make up 
and define a design pattern. However, there is no consensus on the specification 
of the patterns. Tounsi et al. [12,13,15] focused on both the modeling, the formal 
specification and the composition of SOA design patterns [14] and established 
the link between them with an automatic transformation. They used the SoaML 
language for the pattern modeling that ease the understanding of pattern mod- 
els. For the pattern specification, they used the Event-B formal method in order 
to attribute formal notations to SOA design patterns for the purpose of checking 
their design correctness. 

In this work, we are interested with the IoT design patterns. In this con- 
text we find several researchers who proposed a set of IoT design patterns in 
various categories. Eloranta et al. [3] proposed patterns for the construction of 
distributed control systems. Qanbari et al. [9] presented four patterns for the sup- 
ply, deployment, orchestration and monitoring shipboard applications. Lukas et 
al. [10,11] have published patterns for device power supply, operation and com- 
munication modes and a number of IoT design models. All these patterns are 
described with a visual and informal notation. There is no formal semantics asso- 
ciated. There are few numbers of research work that deals with the modeling of 
IoT patterns. Borelli et al. [2] proposed a language called BIoTA (Buildout IoT 
Application Language), to assist and streamline the building of software archi- 
tectures for IoT. The specification and implementation of the BIoTA language 
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involve a grammar and a compiler, responsible for syntax and semantic analy- 
sis, as well as code generation. This work facilitate the formalization of software 
architectures using automata, which would otherwise be created in an informal 
and ad-hoc manner but it addresses specific IoT scenarios. In this paper, we 
present the formal modeling of IoT patterns proposed by Reinfurt et al. [10]. 


3 Pattern Specification 


Ensuring the reliability and the correctness of IoT design patterns is a goal that 
we have fixed. For this, we propose an approach to formally specify IoT design 
patterns by using the formal method Event-B that is well suited to our needs 
and goals. This approach allows the validation of the modeling part and ensures 
the verification of the relevant properties of design patterns. 

Event-B is a formal method for system-level modelling and analysis [1]. It 
is well-suited for specifying IoT design patterns because: The primary concept 
in doing formal developments in Event-B is that of a model. It is made of sev- 
eral components of two kinds: machines and contexts. Machines contain the 
dynamic parts of a model, whereas contexts contain the static parts of a model. 
'Thanks to this classification, Event-B allows the specification of structural and 
behavioral features of design patterns. Refinement techniques proposed by this 
method allow us to build patterns gradually and at different abstraction levels. 
Mathematical proofs allow verifying model consistency and consistency between 
refinement levels. The most important reason to use Event-B method is the 
availability of a supporting tool called the Rodin platform for modeling and 
automated proof [1]. The platform is open source and is further extendable with 
plug-ins. A range of plug-ins have already been developed including ones that 
support animation and model checking like the Prob plug-in [8] that we used. 

There are two general pattern models as instances of the proposed metata- 
model depending on the location of the device. If the device is placed locally, we 
use the ^Medium Based Bootstrap Pattern" as a solution to configure the new 
device. If it is placed at a distance, we use the “Remote Bootstrap Pattern” as a 
solution. 

In the specification, we concentrate on two categories of design patterns; 
“Bootstrapping Design Patterns" and “Registration Design Patterns". “Boot- 
strapping Design Patterns" allow configuring new devices. They are composed of 
“Medium Based Bootstrap Pattern" and “Remote Bootstrap Pattern". “Medium 
Based Bootstrap Pattern” allows to configure a new device on-site through a 
removable storage medium inserted in the device. This support contains the 
necessary information for configuration. “Remote Bootstrap Pattern” is a con- 
figuration pattern used in case that a device is placed far away and is difficult 
to reach. The configuration in this case is done by downloading configuration 
information from a bootstrap server. 

“Registration Design Patterns” allow to register the attributes and the fea- 
tures of a new device on the Back-end server. The registration is used to facilitate 
the communication and the interrogation with other connected objects. There 
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are many registration patterns. In this work, we present two patterns. So “Regis- 
tration Design Patterns” are composed of “Automatic Client Driven Registration 
Pattern” and “Server Driven Model Pattern”. The “Automatic Client Driven Reg- 
istration Pattern” allows the device to register on the Back-end server via an API 
call. The “Server Driven Model Pattern” is used to create a device model that 
includes its description and functionality. 

We model structural features of design patterns with contexts in the Event-B 
method. They describe by a set of concepts, the structure of an IoT architecture. 
We use it to describe the architecture of IoT design patterns. More specifically, 
it is to define the entities that can be involved in the pattern, their types and 
their dependencies (connections). We model behavioral features of design pat- 
terns with machines in the Event-B method. We describe through this Event-B 
component successive interactions between the different entities of the Io T appli- 
cation in order to represent the two categories of the design patterns. 


3.1 Contexts and Machines Relationships 


We present a specification of two categories of patterns. So, the specification 
will be too complicated and error prone if it is done in one shot. To reduce 
this complexity, we define specification levels. At the first level, we create a 
very abstract model (a context CO and a machine M0). At the next levels, we 
use refinement techniques to gradually introduce detail and complexity into our 
model until obtaining the final pattern's specifications. Our refinement strategy 
is explained in Fig. 1. First, we create a generic context C0. This generic context, 
presented in follows, plays the role of a metamodel to define types of IoT design 
pattern elements. 

At level 1, we present contexts of the pattern category in general with their 
properties. These contexts extend the generic context. We specify in the con- 
text “BootstrappingPatterns”, which extends the “C0” context, the set of con- 
figuration patterns contained in the ^MeduimBasedBootstrap" pattern and the 
“Remote Bootstrap” pattern. We also declare the location to specify the posi- 
tion of a device (Device) if it is locally located or remotely. In the context 
"RegistrationPatterns" which extends the context “CO”, we specify that in the 
"RegistrationDesignPatterns" category, we find the *AutomaticClientDrivenReg- 
istration” and “ServerDrivenModel” patterns. At level 2, we present contexts of 
the different patterns belonging to these categories as an extension of their pat- 
tern category. These contexts are used by the corresponding machines in order 
to specify their appropriate behaviors. 


3.2 Structural Features 


'There are two major types of entities in the Internet of Things. The first is an 
«Object» or a “thing” which you intend to make smart by providing connectivity. 
'The other is the « Component» or embedded system which provides this connec- 
tivity. Sensors, Actuators, Smart Things, and Devices are examples of Objects. 
Using Event-B, we specify in the context C0 the two entities as constants. The 
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Fig. 1. Contexts and machines relationships 


set Entity is composed of the set of all Components and the set of all Objects 
(Entity = ComponentUObject ^ Componentn Object = €). This is specified 
by using a partition in the AXIOMS clause (Entity partition). 


SETS 
Entity 
CONSTANTS 
Component 
Object 
AXIOMS 
Entity partition : partition(Entity, Component, Object) 


Components name C; are specified as constants in the CONSTANTS clause. 
Formally, we specify (C4,...,C4,] € Component ^ C1 #C2 ^...^ Cn-1# Cn. 

Objects name O; are also specified as constants. Formally, we specify 
{O1,...,0n} € Object ^ O1 Z O2 A...A O4 1 On. 


CONSTANTS CONSTANTS 
Ci O1 
Cn On 
AXIOMS AXIOMS 
Component TYPE: {Ci,...,Cn} Object TYPE: {01,...,On} 
€ Component € Object 
axm 1: C41 #C2A...A axm 1: 01 Æ O2 ^ ...^ 
Cn-1 £ Cn. On-1 # On. 
The communication path between Entities within an architecture is called a 


«Connector». It ensures the link between a «Provided» port and a « Required» 
port to form a complete and coherent system. «Ports» constitute the interaction 
points with entities environment. A required interface on a port specifies one or 
more operations required by behaviors of the object. A provided interface on a 
port specifies one or more operations that an object must provide. 
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A «Connector» PushE;E; is a link between two entities. It can be between 
two components (PushC;C;), two objects (PushO;O;) or between a component 
and an object. When the direction of the connection is from a component to an 
object, it is named PushC;O; and if it is from an object to a component, it is 
named PushO;C;. Formally, Connectors are specified with an Event-B relation 
between two entities. Connector's name PushE;E; are specified with constants 
in the CONSTANTS clause. The set of Connectors is composed of all Connectors 
name. This is specified formally with a partition (Connector partition). 


CONSTANTS 
Connector 
PushE; Ej 


Push E, Em 

AXIOMS 
Connector Relation : Connector € Entity = Entity 
Connector partition : partition(Connector, {Push E; Ej), ..., 


(Push E, Em}) 


To define the source and the target of a connector, two axioms must be added, 
namely the domain and the range. 


PushE;E; Domain : dom({ PushE;E 
} 


E jt {Ei} 
PushE;E; Range: ran({PushE; Ej } 


j= {5; 


Internet of Things (IoT) architecture requires a message-based communica- 
tion. Formally, «MessageType» is the type of messages exchanged between dif- 
ferent entities, it is declared in the SETS clause. Messages name M; are specified 
in the CONSTANTS clause. They are attributed with their type with a partition 
in the AXIOMS clause (Message partition). 


SETS 
MessageT ype 

CONSTANTS 
Mi 


Mn 
AXIOMS 
Message partition : partition(MessageT ype, {M1},...,{Mn}) 


3.3 Behavioral Features 


A machine of a pattern specification Mi has a state defined by means of a 
number of variables and invariants. Some variables can be general as the variable 
Send, which denotes the sent message. The state can be modified by means of 
events. Events are instantaneous and their effect can cause the occurrence of 
other events. This copes well with the behavior of IoT design patterns. The 
sending of a message is instantaneous and thus can lead to the sending of other 
messages. 

The structure of a machine PMi is presented in follows. Machine PMi sees 
context PCi. Variables V1 are defined to know the availability of entities, sent 
messages, processed messages, etc. Invariants INV1(V1, S, X, Y) are defined to 
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specify the various predicates which the variables must obey. Events (Evt1.1, 
Evt1.2, ...) are the various events of the machine. 


MACHINE PMi 
Sees 
PCi 
Variables 
V1 
Invariants 
INV1(V1,S, X,Y) 
Events 
Evt1.1, Evt1.2, ... 
END 


Entity activation is specified with a variable Dispo in the VARIABLES 
clause. It is used in order to know the availability of each entity (available or 
not). Thus, it is declared with a partial function between the Entity type and a 
the Boolean type. This is expressed with the invariant Dispo Function. 


VARIABLES 
Dispo 
INVARIANTS 
Dispo Function: Dispo € Entity + BOOL 


Internet of Things (IoT) architecture requires a different kind of message 
queue based communication than other types of software systems or big data 
solutions. With IoT, the messaging needs are more complex since IoT requires 
two way message communication between the server-side or cloud components 
and the IoT hardware devices. Formally, the variable Send is defined with 
the invariant Send Relation which specify that Send is a relation between a 
Connectors and a MessageT ype. In this way, we know how is the sender, the 
receiver and the sent message. 


VARIABLES 
Send 
INVARIANTS 
Send Relation : Send € Connector = MessageT ype 


Each pattern has its own behavior but some events can be general like the 
event of sending a message Sending M;. 


Event Sending M; 
when E 
grd : G(v) 
then 
act : Send := Send U (Push E; E; — Mi) 
end 


If the transition is reflexive, the event includes an action of the form: 


Var Name:— Var Name < (Entity Name = M;} 


For example, the transition of message processing is a reflexive transition. Then 
the event Processing M; includes the following action: Process :— Process + 
[O;— Mil. That is, object O; processes message Mj. 
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3.4 Formal Verification 


During our development, we use refinement techniques, they consist on devel- 
oping a series of more and more accurate models of the pattern we build [1]. 
By using refinement techniques, we enrich the pattern model, and we make sure 
that refined models are not contradictory. As a result, when the last model is 
finished, we will be able to say that this model is correct by construction [1]. 

Four formal verification steps have been developed for checking design pat- 
terns; type checking, model checking, animation and theorem proving. Type 
checking is a technique controlling low level properties of variables in a specifi- 
cation. It is achieved within the compiler. Model checking and animation are two 
techniques used to show the dynamic behavior of a model, and they allow one to 
systematically exploring all its reachable states. We use them to check the cor- 
rectness of the behavior of the pattern. Some temporal/behavioral properties are 
verified like liveness (no deadlocks present in the model) and reachability (prove 
that an event whose guard is not necessarily true now will nevertheless certainly 
occur within a certain finite time) properties. This is achieved by the model 
checker ProB [8]. Theorem proving technique allows checking properties which 
can be experimented either as predicates (INVARIANTS, AXIOMS, THEOREMS) or 
with guards in the events. It is also ensured by proof obligations generated by 
the Rodin platform. They define what is to be proved to ensure the consistency 
of an Event-B pattern model. As example of consistency constraints, we check 
that no entity can send a message if it is not authorised. This is controlled by an 
invariant called Can_Send_INV. For sequence diagrams, we require that every 
message must start activation. 


Vz,c,y:z € Entity ^ (x — y) € Connector + MessageT ype 
Adom({x}) = {z} ^ x = y € Send > z ^ y € Can Send 


Our approach ensures that the specified IoT design patterns are correct by con- 
struction. It offers architects the ability to reuse correct pattern models, which 
saves their efforts to prove the correctness of the patterns. 


4 Case Study: Smart Hospital 


Smart hospitals are hospitals which build new infrastructure. In this infrastruc- 
ture, all are enabled by underlying digitized networking infrastructure of inter- 
connected assets and remote-controlled automated components which can be 
medical sensors, medical devices, etc. It can include several other components 
that can be monitored and controlled remotely to offer providers a continu- 
ous stream of real-time health data such as heart rate, blood pressure, oxygen 
concentration level, and glucose monitoring. Most of these components can be 
controlled by a mobile device or a computer. In our case study, we add a new 
device (a camera) to a smart hospital patient room. This device makes it possible 
to control the various rooms of the hospital for safety issues. For example, the 
camera can recognize when a patient sits up or even has a restless night's sleep. 
'Then it informs the nurse immediately through a notification sent to their smart 
phone. It can also send alerts when it detects big problems. First, the camera is 
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Fig. 2. Smart hospital case study 


added without any information to initiate its first connection. We then apply the 
Medium Based Bootstrap design pattern to have its configuration information. 
This information is inserted into the device through a memory card. Second, 
we go through the registration procedure on the main server (BackEndServer). 
This procedure is done through the use of the two patterns of the Registration 
Design Patterns category which are the automatic client driven registration pat- 
tern and the server driven model pattern which allow registering the device on 
the main server. Finally, the camera became able to communicate and create 
connection links with its communication partners. The camera communicates 
with the nurse's smartphone to notify him of what is happening in real-time. 

We model this application through the use of the model shown in Fig. 2. The 
camera is associated with an object of type ^Device", the memory card is defined 
as an object of type “Storage Medium” and the Smart phone is associated as an 
object of type “SmartThing”. The propagation of events between objects is done 
through a DeviceGateway. 

In Fig. 3, we specify the static part of the design patterns of the model applied 
in the Smart-Hospital case study using contexts. We first specify the generic 
context C0. We change the generic concepts previously defined by the concepts 
of our case. A Camera is a Device of type object. A Smart phone is a Smart Thing 
of type object and a Memory card is a Storage Medium of type object. Then we 
specify the other contexts belonging to the Bootstrapping and the Registration 
patterns. In Fig.4, we specify the dynamic aspect in the machine part of the 
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CONTEXT 
C0 
SETS 
Entity 
MessageType 
CONSTANTS 
Object 
Component 
Camera 
SmartPhone 
Connector 
AXIOMS 


axml: partition(Entity, Component, Object) 


axm2: Camera € Object 

axm3: SmartPhone € Object 

axm4: SmartPhone # Camera 

axm: Connector € Entity > Entity 
END 


CONTEXT 


EXTENDS 
BootstrappingPatterns 
CONSTANTS 
MemoryCard 
AXIOMS 
axml: MemoryCard € Object 
axm2 : MemoryCard z Camera 


axm4: Location = {local} 


END 


MediumBasedBootstrapPattern 


axm3 : MemoryCard z SmartPhone 


CONTEXT 
BootstrappingPatterns 
EXTENDS 
CO 
SETS 
Location 
CONSTANTS 
remote 
local 
AXIOMS 
axm1: partition (Location, {local}, {remote}) 
END 


CONTEXT 
AutomaticClientDrivenRegistrationPattern 
EXTENDS 


RegistrationPatterns 
CONSTANTS 

BackEndServer 

API 

PushAR 

PushAD 

PushDA 
AXIOMS 
axm1: BackEndServer € Component 
axm2: API € Component 
axm3: BackEndServer 4 APT 
axm4: Connector = {PushAR, PushAD, PushDA} 
axm5: dom({PushAR})={API} 
axmó: ran({PushAR})={RegistryDevice} 
axm7: dom({PushAD})={API} 
axmó: ran({PushAD})={Camera} 


END 


Fig. 3. Contexts belonging to the case study 
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Event-B method. We specify a part of the behavior of objects constituting the 
smart hospital model of our case. We describe this behavior of the configuration 
in the machine named BootstrappingPatterns. At the end of the specification 
Event-B models are proved. This case study shows that our approach is able to 
be applied while also taking the quality of results and performance into account. 
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MACHINE Event SendInformation: 
BootstrappingPatterns WHERE 
SEES grd1: Localization = remote 
MeduimBasedBootstrapPattern grd2: Configuration = FALSE 
RemoteBootstrapPattern THEN 
VARIABLES act]: SendMessage := SendMessage U 
SendMessage { PushBD + Information } 
Configuration act2: SendResp :— SendResp U 
ACK { PushDB » ACK j 
SendResp act3: Configuration := TRUE 
Information END 
Pastelnfo 
INVARIANTS Event PasteInformation: 
inv1: SendMessage € Connector > MessageType WHERE 
inv2: Configuration € BOOL grd1: Configuration = FALSE 
inv3: Information € MessageType grd2: Localization — local 
inv4: SendRep € Connector +> MessageType THEN 
inv5: PasteInfo € Object €» Object act] :PasteInfo ‘= PasteInfo U 
EVENTS {StorageMeduim » Device] 
INITIALISATION: act2: Configuration = TRUE 
THEN END 


actl: SendMessage := Ø 
act2: Configuration := FALSE 
act3: SendResp :— Ø 
act4: Pastelnfo :— Ø 

END 


Fig. 4. Machine BootstrappingPatterns belonging to the case study 


5 Conclusions 


In this paper, we presented an approach that allows to formally specify connected 
object architecture design patterns. In particular, specifying the “Bootstrapping 
Design Patterns” category and the “Registration Design Patterns” category. We 
formally specified these design patterns using the formal Event-B method and we 
described their structural and behavioral features. In our stage, we are working 
at a high level of abstraction, so there is no connection between the theoretical 
model and physical devices for machine learning. In our current work, we are 
working in it. The generalization of our approach is an objective that we have 
fixed. In previous work [7] our approach was validated with a Smart Home case 
study. In this paper, we applied our approach in a health care case study. As 
future work, we are working on applying our approach on other categories of IoT 
design patterns. 
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Abstract. The rapid advancement of the Internet of Things (IoT) has reshaped 
the industrial system, agricultural system, healthcare systems, and even our daily 
livelihoods, as the number of IoT applications is surging in these fields. Still, 
numerous challenges are imposed when putting in place such technology at large 
scale. In a system of millions of connected devices, operating each one of them 
manually is impossible, making IoT platforms unmaintainable. In this study, we 
present our attempt to achieve the autonomy of IoT infrastructure by building a 
platform that targets a dynamic and quick Plug and Play (PnP) deployment of 
the system at any given location, using predefined pipelines. The platform also 
supports real-time data processing, which enables the users to have reliable and 
real-time data visualization in a dynamic dashboard. 


Keywords: IoT - Dynamic deployment - Plug and play - Real-time processing 


1 Introduction 


Enterprise architectures with a large number of the Internet of Things (IoT) devices 
impose challenging operational issues. Itis possible to deal with these issues for a limited 
number of IoT devices, yet it is almost impossible to do likewise for many devices. We 
argue that a faster approach to tackle the operational issues of IoT by adopting the Plug 
and Play (PnP) concept in IoT architecture. IoT is a concept that builds a connection 
between all devices to the internet and facilitates communication with each other over 
the internet, with over 10 billion IoT devices wirelessly connected as of recently [1, 2]. 
In today's IoT technology, the major requirement is an architecture which is capable of 
supporting a large number of devices and still provides reliable performance. With the 
increasing number of devices, it becomes exceedingly complex to add a new one to an 
existing platform, and to reduce the integration phase of the system, therefore the concept 
of PnP in asmart environment was first proposed in 2006 [2]. The main objectives of PnP 
architecture in IoT systems are to reduce user intervention for system integration, and to 
have a real time solution from the system services. As an example, Helal et al. proposed 
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a platform based on the PnP principle that enables a seamless integration of devices to 
the back-end systems without exterior interventions from integrators or engineers [3]. 
Although the concept of PnP is very intriguing, the PnP based IoT systems’ reliability 
is the next big challenge. The reliability of a system can be evaluated by: the ability 
to perform a required task, ability to perform without failures, ability to perform under 
stated conditions, and ability to perform for a specified period [4]. 

The rest of the paper is organized as follows. Section | is the literature review of the 
existing architectures, implementation of IoT platforms, and visualization techniques. 
In Sect. 2, we propose our AMI IoT platform, wherein we give an overview of our 
architecture and implementations. In Sect. 3, we move into the details of our architecture 
and implementations of different components of the PnP base IoT architecture. In Sect. 4, 
we have the results and validation of our platform. We finally conclude the paper with a 
summary of our platform and its uniqueness compared to the other existing solutions in 
Sect. 5. 


2 Literature Review 


The reliability of PnP in IoT systems is linked to the diverse components of the IoT 
architecture. Therefore, there is a need to understand the different architectural imple- 
mentations for IoT platforms that cover the existing domains. In the following, we discuss 
the existing systems and their architectural implementations. 


IoT Platforms and Architectures The IoT architectures that exist today have an evolv- 
ing pattern, each new one overcomes the major challenges faced by the previous one. 
The initial approach was the three-layered architecture (Fig. 1.a) [5] with a limitation of 
having only the cloud layer for data processing, then came the Fog layer architecture, 
which resolved the limitations of cloud computing, by extending cloud-based services 
and bringing them closer to the IoT end-user environment, but it still had many issues 
on every layer, each distinguished by specific faults and errors that lead to an overall 
infective IoT system: * First, starting from the end-user layer, where the sensors and/or 
actuators recording environmental data could have reading errors [6]. * Second, the 
fog layer, where the gateways face a lack of memory, and processor overloading and 
overheating, which leads to serious issues in IoT systems [6, 7]. * Third, the network 
layer where communication links in the IoT system are frequently prone to breaking, 
malfunctioning (e.g., incorrect state), or unavailability of the medium [7]. * Fourth, the 
cloud layer with the challenge of sustaining all the different connected gateways, on 
both the computation and maintenance level. 

We run a generalized analysis on the existing IoT platforms based on the following 
metrics- the deployment duration, the real-time capability, the reliability of the system, 
the business layer (mainly the data visualization aspect), and finally the cost factor, and 
following are the results. The existing platforms can be classified into two major groups 
* Consumer IoT (e.g., OpenHAB [8], SmartThings [9], HomeKit [10]) and * Enter- 
prise IoT (e.g., AWS IoT [11], Watson IoT [12]), the dependencies of this classification 
include functionality and contextual information, interoperability, scalability, security, 
and costs [13]. We can categorize the existing platforms into three sets. * The first set of 
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(a) (b) (c) 
Fig. 1. IoT (a) three-layered, (b) five-layered, and (c) Fog/IoT architectures [5] 


the platform involves a centralized hub (e.g., OpenHAB or SmartThings). The advan- 
tage is that it provides data processing and storage in the local settings meaning fast data 
processing and real-time visualization along with offline capabilities. The disadvantages 
include higher resource requirements, higher cost of the entire setup. This implemen- 
tation also involves a lot of user intervention for the configuration of the system which 
hinders its scalability factor and in turn does not support the PnP aspect of IoT systems. 
* The second set, which involves the cloud implementations, avoids the initial involve- 
ment of the users to a greater extent along with no requirements of any specific hubs 
which reduces the cost (e.g., AWS IoT, Watson IoT). The major drawback is the real-time 
performance or the reliability of the system as it depends on the connectivity of the sys- 
tems. *The third set is the hybrid approach which uses most of the hub implementations 
and the cloud services. Watson IoT or AWS IoT or SmartThings have tried optimizing 
the entire platform pipeline by segregating them into the local edge processing and the 
cloud processing, but we still find issues with deployments or installations due to the 
complexities of having to configure the environmental settings. The other major issue is 
the cost of the systems, which are often not affordable to all users. 


Plug and Play in IoT Platforms In order to overcome the abovementioned issues, sev- 
eral studies draw their attention to bring the idea of PnP into IoT platforms. Kim et al. 
[14] proposed using web semantics to manage IoT devices. In their approach all sensors 
and actuators are connected to Arduino boards as embedded IoT devices. Each Arduino 
can accept configuration from central server and adapt their functionality to the new con- 
figuration. Meanwhile, they developed a Knowledge Base which stores the configuration 
of all supported devices and once a new device is connected, appropriate configuration 
is shared with the device so that they can start collecting data or triggering actions to 
control the environment. As a result, they were able to reduce the complexity of deploy- 
ments and provide more dynamicity in IoT environments. However, they only focused 
on the perception layer and did not address the business and application layer, and how to 
visualize the result dynamically. *) Kim et al. [15] target a framework dynamic enough 
to integrate new sensors, devices or services to an environment. They proposed a device 
discovery service that (1) automatically detect and generate a description of the device 
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and its functionalities, and (2) integrating that same device seamlessly in the reasoning 
process without causing any interruption to platform or the framework functionality. 
*) Kesavan et al., bring forward the scalability and dynamic approach which is an essen- 
tial aspect for remote monitoring of patient data with real time analysis [16]. In their 
proposed method they focused on the scalability of the system in accordance with the 
PnP approach. The gateway is capable of handling dynamic addition of sensors and 
is capable of handling data transmission in a seamless manner. Another aspect of this 
method is the edge analytics part which informs the users of the battery level status of 
the sensors used so that necessary actions can be taken when required. 


3 AMI Platform 


To address all the drawbacks of the implementations described above, we built AMI 
platform, based on the fog layered architecture (Fig. 1.c). AMI platform architecture 
has five main layers (Fig. 2), the edge layer which includes all the sensing devices with 
different communication protocols, the fog layer which is tasked with preprocessing the 
acquired data before sending it to the above layers, then comes the network layer which 
deals with the communication protocols and the security and privacy of the data being 
transmitted to the cloud. The fourth layer is the cloud layer, where the data received is 
decoded and processed in specific models which are then stored in storage hubs. The 
final layer is the business layer where the real-time data is visualized in a dynamic pattern 
through dashboards based on the user specification. Our platform targets supporting PnP 
features in all the presented layers, and as well as the link between them. 
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Fig. 2. AMI IoT architecture 
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3.1 Architecture Overview 


The AMI Platform involves a set of physical (i.e., sensing devices and computing hard- 
ware) and logical (i.e., software applications and virtual computing resources) com- 
ponents that collect, process, transmit, store, and analyze data. These components are 
layered following Fog Computing architecture. 

The physical gateway is situated in the end-user environment. It represents the fog 
layer and thus the bridge connecting the sensor devices in the end-user layer with the 
network layer. Additionally, a virtual gateway is located inside the cloud infrastructure 
as an extension of the network layer, to receive the data transmitted through the network 
layer, and push it to the cloud storage. The benefit of the separation is to utilize the 
computing resources on the gateways for pre-processing and ensuring the quality of 
service. A common challenge with the physical gateways is the deployment and update 
of the numerous services that are running on them that handle the communication and 
acquisition of data from sensors. Our focus while designing and building the physical 
gateway was to ensure the ease of PnP integration of a "Pipeline" that automatically 
or semi-automatically handles the deployment of said services. The data flows through 
the platform as follows: First, at the end-user layer, a cluster of heterogeneous sensors 
collects both environmental (e.g., temperature, motion, location) and biomedical (e.g., 
heart rate, breathing rate, sleep pattern) data. Next, the data is sent to the physical 
gateway for pre-processing and pushed to the cloud layer via a secure channel created 
by the network layer. Upon arriving at the cloud layer, the data will be decoded, processed 
in specific formats, and stored in databases. The business layer is for the user interface 
which also has models for dynamic graphs and charts creation which helps for a better 
adaptation of data visualization for any kind of user need. We also propose a different 
approach to resolve the existing challenges of scalability and maintainability based on 
microservices, where the different functional components of the system are split into 
independent applications [17]. 


3.22 Edge Layer 


In the context of IoT, the end-user layer is located in the monitored person's living 
environment, and it is considered the entry point of the platform. It consists of a set 
of sensing devices of heterogeneous nature that observe key parameters of both the 
living environment and the person in multiple dimensions. *) Wearable Sensors (e.g., 
Apple Watch, Garmin Watch, Fitbit Watch, Mi Bands, Pulse Oximeters), which are 
installed and/or implanted on participants' bodies (e.g., wrist, fingers) to monitor their 
everyday activities, provide various measurements, including motion (e.g., acceleration, 
step count, distance walking), location (e.g., attitude, longitude, altitude), biomedical 
information (e.g., heart rate, heart rate viability, breathing rate, and oxygen saturation). 
One major advantage of wearable sensors is their portability: the participants can eas- 
ily carry the devices either at their home, doing outdoor activities, or during clinical 
assessment. However, this type of sensor also shows limitations under specific circum- 
stances. Because the patient's expertise in advanced technologies cannot be guaranteed, 
ensuring proper charging, usage, and maintenance of the devices are challenging tasks. 
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*) To overcome these limitations, there is the second type of sensing devices, Environ- 
mental Sensors (e.g., door/window, motion, temperature, illuminance, and humidity 
sensors) that are designed to track indoor environmental parameters and patients’ ADL 
continuously while maintaining their autonomy and independence [18]. These sensors 
are often installed in fixed positions (e.g., wall, door, and window) inside a subjective 
room. *) Additionally, other sensors can be placed or connected to furniture such as 
the smart mat device, a highly sensitive micro-bend fiber optic sensor (MFOS) often 
placed on top of the bed (under the mattress or bed sheet) or on a char non-intrusively to 
monitor location information and vital signs (e.g., heart rate (HR), breathing rate (BR), 
Ballistocardiograph (BCG)), which enables the possibility of detecting sleep cycles and 
sleep disorders, such as Apnea [19]. 


3.3 Fog Layer 


Our implementation of the fog layer is represented as a physical gateway, that acts as the 
hub for multiple devices with different communication protocols. The physical gateway 
has the following major functionalities: *) Data acquisition: the gateway supports diverse 
communication protocols, e.g., Z-Wave, Insteon, Bluetooth, Serial line communications, 
and other similar protocols. Each adapter is an abstract layer for the hardware and 
constantly listens to its corresponding sensor. It also enables easy and reliable integration 
of new devices (PnP concept in sensor installation [20]) with the same set of protocols. 
*) Preprocessing of data: the raw data obtained from heterogeneous sensors suffers 
from poor interoperability, so the data format varies between different types of sensors, 
leading to difficulties in data interpretation and transmission. Therefore, we developed a 
data model library to overcome this issue. The data model library enables the gateway to 
pre-process the data by adding semantic information and populating its values into one 
of the compatible formats. *) Data Transfer: The next step is the transfer of data from the 
edge layer to the network layer, the processed data is sent through network protocols, for 
example, the messaging protocol called Message Queue Telemetry Transport (MQTT) 
or the hypertext transfer protocol (HTTP) protocol. Currently, in our implementation, 
we are using MQTT as it has a publish-and-subscribe architecture where the client 
devices and applications publish and subscribe to topics handled by a broker. *) Status 
Updates: The other service offered by the gateway is the system or status updates. All 
the above-mentioned services require system resources and to avoid failures the system 
check services provide live status updates of the resources, for example, CPU, memory, 
and internet connectivity. 


3.4 Network Layer 


Sending confidential data and information over the internet is vulnerable to attacks, so it 
is critical to create a secure channel for connections, which is taken care of by the network 
layer, the subspace between the physical gateway (at the edge of the end-user layer) and 
the virtual gateway (at the edge of the cloud layer). A great amount of effort is put into 
securing the network connection. *) Firstly, a virtual network is created to establish a 
peer-to-peer secure channel between the two gateways, adding an extra layer of security 
to our protocol of communication. The advantages of the software network solution are 
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efficiency and quick adaptability. It takes advantage of the existing internet connection 
instead of setting up a new one. In our platform, we use OpenVPN to implement this 
secure channel. OpenVPN is an open-source virtual private network (VPN) system 
that creates routing and bridging connections between entities. An OpenVPN server 
automatically issues certificates for our physical and virtual gateways, triggered by their 
respective deployment pipelines, and then stores them in a secure storage hub. This 
approach limits the risks of transporting the certificates/keys from one entity to another 
one. *) lastly, enforce the firewall rule by allowing only a small number of essential 
connections and blocking all others. All communication between the physical gateways 
and virtual gateway goes through this channel, securing the confidential data transmitted 
between them. 


3.5 Cloud Layer 


The cloud layer is responsible for the management, storage of the collected data. Con- 
sidering the essential need for a real-time monitoring solution, the cloud infrastructure 
needs to have minimal delay and downtime, to allow for a reliable and real-time visual- 
ization and analysis. Hence, the agent-based microservices approach is adopted on the 
cloud layer. Each component on the cloud layer is containerized using Docker daemon. 
This allows easy deployment of components anywhere without having to worry about 
the specifics of the underlying platform. 


Virtual Gateway: Being the contact point of the cloud layer, the virtual gateway is 
created from a pre-built docker image and plays a crucial part in the platform. It connects 
with its peer physical gateway through the VPN in the network layer, receives data 
with the configured communication protocols (e.g., MQTT, REST), and reconstructs 
the data message using a mapping function to meet the requirement of the data storage. 
However, managing on the physical gateway the rapid evolution of sensors' data models 
and numerous deployments of physical gateways requires a manual configuration and 
intervention before every deployment and after every update. This challenge pushed us 
to build the gateway in a way that will allow for a PnP integration seamlessly with a 
pipeline that automates its deployments, and updates. 


Data Storage: IoT data representation and storage are an ever-changing and expanding 
field, with this in mind we have opted to use a non-relation database, or more specifically 
document based. The advantages of this are: 1) first the dynamicity and flexibility of 
document-based databases, which is particularly useful for our platform due to the evo- 
lution of data generated by IoT devices over time, and by eliminating the time required 
to adapt or update the structure of tables or other entities to allow inserting new fields or 
data models; 2) second the horizontal scalability of non-relational databases, as opposed 
to vertical scalability of relational databases [21]. As consequence, keeping up with the 
increase in either the volume or velocity is only a question of adding more nodes to the 
database cluster. This architecture also ensures higher availability because, if any node 
fails, all its functions will be delegated to other nodes in that cluster. Our choice of data 
storage is Elasticsearch, an open-source distributed search and analytics engine at the 
center of the Elastic Stack [22]. It supports various commonly used data types, including 
numerical, textual, keywords, state, and multi-dimensional. 
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3.6 Business Layer 


Data Visualization: Environmental and wearable sensors acquire a massive amount of 
data, e.g., on average, 1.2 to 1.8 million entries per day during the course of our first 
deployment. With such amount of data constantly generated, it is crucial to develop 
an appropriate visualization tool that can present collected information in real-time to 
provide better assistance to the end-users. However, the variety of needs, roles, and 
metrics in different applications along with changes in needs over time make it difficult 
to have only one dashboard to fit all the needs. Our adaptive solution enables even users 
without technical knowledge to create a dashboard adapted to their needs in minimal 


time. 
Configuration Tool e Chart Template Designer 


Dashboard Designer 


Fig. 3. Approach for adaptive data visualization 


In our architecture, a unique approach is taken to ensure PnP by visualizing data 
adaptively (Fig. 3). We first choose web-based technologies to create our system due 
to their accessibility. Unlike traditional software that is tied to a specific system, web 
applications are reachable from multiple platforms, including desktop, mobile, tablets, 
and smart TV. Through these web-based solutions, we enable users to dynamically define 
the environment and add sensors to it. When the environment model is defined, they can 
design the dashboard based on their needs which will be adapted to the environment 
automatically. This system consists of three major components: 


1) Chart Template Designer. Empower users to dynamically design charts and save 
them as templates specific to each type of sensors. Users can choose the data source 
and visually generate queries to filter data for charts. Once the user finalizes the chart 
design, the configuration of the designed chart is stored in our data storage, and the 
defined query is parsed to accept parameters in order to enable linking the designed 
charts to external parameters. The first step to design charts is to retrieve required data 
from the data source. We used a query builder tool to allow users to define their intended 
criteria to filter fetched data. We added the concept of chart template to our platform in 
which when a request is sent to view a chart, the configuration of the chart along with 
other information will be loaded. Thus, the designed system gives the user flexibility to 
reuse designed charts in many places by only passing the required parameters (e.g., a 
device id). 

2) Environment Configuration Tool: This tool enables non-technical users to create a 
3D model of the deployment environment, input the address and occupants, and place 
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items, furniture, and predefined sensors using simple drag-and-drop actions [Fig. 4]. 
The output of this component is used by Dashboard Designer and Reasoner Engine to 
generate the intended design and knowledge for each space in this environment. We 
defined the concept of "Space" in our platform as a group of sensors, associated with a 
place. The output generated by this tool is a number of “Spaces,” each of them includes 
several sensors and descriptive information about the area. This form of output enables 
all other consumer components to adapt themselves to the different Spaces. 
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Fig. 4. 3D representation of the environment 


3) Dashboard Designer: Allows users to dynamically design their desired pages accord- 
ing to the needs. This designer incorporates configuration tool data and designed chart 
templates to visualize sensors information on the pages automatically. Users can drag 
and drop necessary components to the page and easily move the elements and change 
components’ size to achieve the desired design. A rich set of components are added to 
the designer which enables users to add and customize pages and link them to environ- 
mental sensors. Once the design is finished, the design will be applied to the all provided 
“Spaces,” received from configuration tool and the data sources for elements in each page 
will be adapted to the related environment automatically. e.g., a user may design a chart 
to visualize room temperature, they can design it for one room and the dashboard engine 
will adapt the chart for other rooms provided by configuration tool and replace all the 
required parameters to visualize data related to each room. Another layer of flexibility 
is provided by allowing users to customize the elements for one specific environment so 
that they can fully adjust the design and deliver the data collected from sensors instantly 
to intended users. To manage the access level of designed pages, we built an integrated 
authentication system based on JWT tokens, each token contains information about the 
user and their assigned roles. Once a user logs into the system, a token is automatically 
added to the requests which allows the server to authenticate users. This authentication 


52 B. Abdulrazak et al. 


service is integrated with our designer to allow specific groups of users to have access 
to certain pages. 


4 Pipelines 


To speed up the process of PnP deploying and running the system, we have created a 
set of pipelines to automate the deployment of all services on every layer of the Fog 
Computing Architecture. It is a three steps process that starts from a sensor-gateway 
connection and ends with a running, real-time dashboard. 


4.1 Gateway 


The gateway acting as the hub is capable of handling multiple sensors simultaneously. 
For the installation of the services to provide the functionalities described in the previous 
section, we use Ansible to automatize the process. The ansible scripts are divided into 
three sections, first group of scripts are tasked with installation of all the required depen- 
dencies that our services would be needing to run properly in the system, along with this 
the setting up of firewall rules and other security aspects are also taken care of by these 
group of scripts, the second section includes the installation scripts for all the services 
provided by our platform, from cloning the repositories to providing them the execution 
rights to act as a system service is being handled in this section. The third section is 
the self-healing installation phase, it installs services that are used for monitoring the 
system resources, and report back for cases of abnormalities. This approach results in 
a significant decrease in user complexity and errors in the installation phase. Once the 
installation is done the gateway nodes are ready to plug and play for any deployment. 
The time taken for this installation phase of the deployment process is approximately 
less than thirty minutes. The gateway nodes have OpenVPN installed during this process, 
which gives them a secure connection to the servers to communicate with. 


4.30 Server 


The server pipeline consists of two main steps, both of which are automated with a script 
as well and require only the identifier of the gateway being deployed for an easy PnP 
solution. Each step corresponds to a layer in the Fog Computing Architecture. The first 
step is the configuration of the Network Layer by registering the gateway certificates in 
the VPN server and then setting up firewall rules to only accommodate the ports used 
by our services, thus protecting the connection between the server and the gateway by 
filtering and blocking all other traffic from gaining unauthorized access. The second 
step is to configure the node microservice that is responsible for receiving and pushing 
the data into our storage hub. As we support multiple communication protocols, the 
microservices need to be configured to a specific protocol before their deployment. 
Once the execution of the deployment script finishes, the gateway would be able to 
connect to our VPN server and start sending the data collected from the sensors and 
actuators to our node microservice, which will then be pushed to our data store. 
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4.3 Dashboard 


This step completes the IoT cycle by offering a plug-and-lay solution to design a dash- 
board where we can monitor the collected data in real-time. The first step is to design the 
Environment in our 3D configuration tool where we would create the 3D space where 
we have deployed our gateway and communicate with it to collect the list of sensors and 
actuators that have been deployed in the environment. Right after we feed the generated 
Environment Model to our Dashboard Designer to automatically generate a view for all 
the spaces in the environment and the devices that have been deployed in each space. 
The user has full flexibility to design the required pages. 


5 Results and Analysis/Validation 


Our platform was deployed for a project which validates the working of our architectural 
metrics. The project was conducted for a span of six months at an apartment in AMI. The 
main goal of the project was to monitor the behavior change of a person based on the 
type of medication received at different stages of medical intervention. For the platform 
analysis, the metrics we targeted in this deployment are as follows: 


5.1 Fast Deployment 


We deployed in an apartment to monitor the behavior of a person and to mark the changes 
with the change of medication provided to that person. In this deployment, we deployed 
two nodes (Raspberry PIs), five door sensors, and seven motion sensors, along with 
a sleep mat. The steps followed in the deployment of our platform in the apartment 
are as follows: scouting of the area to have a better understanding of the environment, 
start connecting our nodes to the power outlets and configuring them with the network 
available in the apartment, once the node configuration is done with the network we 
start attaching the sensors at different places in the apartment like the walls and doors, 
with the completion of our entire setup the final step is to check the system performance 
and the data being transmitted to our servers. This entire deployment, from attaching 
the sensors to the walls and doors in every bedrooms and Kitchen and living room, to 
deploying the nodes to set up the network for the nodes, took us less than two hours in 
the apartment. 


5.2 System Reliability 


The key challenge in any IoT system is the reliability of the platform. Reliability in 
this context refers to system failures and the efficiency of the platform in providing 
the decided upon services through various technical and non-technical challenges. The 
technical challenges include the resources of the system, the CPU and memory capacities 
might fail or overflow with time resulting in an unreliable system. The non-technical 
challenges include internet failures, the offline capability of a system is a big challenge 
in the development of any IoT system nowadays. In our deployment, we received certain 
reporting on our system status (Fig. 5) which highlighted the warnings related to CPU 
usage, physical memory usage, internet connectivity, and service resource consumption. 
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Fig. 5. Treemap of all the system status. 


5.3 Real-Time Data Processing 


In the healthcare system, a real-time platform is essential because of the impact each 
data might contain. For example, heart rate monitoring systems cannot have a huge delay 
for patients with cardiac issues. In IoT, there are very few systems in the market that can 
provide a real-time data processing platform. The deployment of our platform was at 
an apartment in Sherbrooke, Canada, which is around 10 kms (about 6.21 mi) from our 
servers. Our architecture has proven to be able to provide a real-time data processing 
and visualization technique with an average delay of less than one second (Fig. 6). We 
have a specific model for the data encapsulation in which we have two timestamps, the 
first timestamp (phenomenonTime.instant in Fig. 6) is recorded at the moment the data 
is received from the sensors and the other timestamp (resultTime in Fig. 6) is noted after 
the data is processed in the gateway and is ready to be transmitted. Based on these two 
timestamps we see that there is very little to no delay being added in our data processing 
period. 


-id RaspberryPi/DCA63294CC84/Serial_Line/ttyS@/JuvoSleep/98D8632B38A8/Amplif 
ied Raw.Data/5/2021-12-02T22:11:14.661-05:00 


-index om-dca63294cc84-2021-07-27 
-score - 
-type -doc 


device.cid RaspberryPi/DCA63294CC84/Serial Line/ttyS0/JuvoSleep/98D8632838A8/Amplif 
ied Raw.Data/5 


id RaspberryPi/DCA63294CC84/Serial_Line/ttyS@/JuvoSleep/98D8632B38A8 /Amplif 
ied Raw Data/5/2021-12-02T22:11:14.661-05:08 


phenomenonTime.instant Dec 2, 2021 6 22:11:14.661 -05:00 


result.valueNumeric 1,403 


resultTime Dec 2, 2021 6 22:11:17.063 -05:00 


Fig. 6. A JSON of stored sensor reading 
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6 Discussion: Lessons Learned 


The results obtained with the deployment of our platform showcase our real-time success 
however we faced a few major challenges throughout the period of deployment. The 
first major challenge was the sensors or device malfunctioning. Now our platform is 
dependent on third-party manufacturers for the devices and the sensors which hinder the 
performance of our system. One of the deployed sensors stopped working due to some 
manufacturing issues in the very first week of deployment. We quickly resolved the issue, 
by changing the sensor with a new one, as our system has a continuous monitoring aspect 
which informed us about the failure in a short time interval. Using trusted manufacturers 
for the devices might lead to higher prices but comes with greater reliability. The second 
challenge was with memory leaks. Currently, the physical gateways (in our case, the 
raspberry pi) have a limited memory resource. Due to the explicit number of services 
running there was amemory leak concerning the logging of issues. This issue was quickly 
resolved with the introduction of another self-monitoring service which monitored the 
overall memory usage of the system along with specific usages by each service. If the 
usage surpasses a specific threshold the logs are rotated and the unwanted system logs are 
deleted. Along with this our visualization platform provided a great level of flexibility 
and enabled us to design desired content for our deployments. However, there are still 
spaces left for improvement, especially when users add a large number of visualizations 
and charts. The high-frequency data generated by some devices (namely smart mats) 
was another challenge for visualization which we were able to overcome in the new 
versions of our platform. For our future works, we have already advanced on some new 
self-healing concepts [23] and distributed architecture to provide better performance 
while maintaining the quality of service (QoS) of our IoT platform. 


7 Conclusion 


Faster PnP deployments of IoT platforms with reliable performance, especially in tack- 
ling real-time data processing, is a challenge that hinders wider implementation of most 
IoT solutions. Thus, we focus in this paper on the approach to addressing these key issues. 
We start this paper by reviewing the progress of the research works related to IoT archi- 
tectures and platforms that are in the market. After analyzing the existing solution, we 
come across the definitive drawbacks, and we present our architecture and IoT platform 
which addresses these drawbacks. We also detailed in this paper each of the components 
of the architecture and described how components are connected to overcome the said 
drawbacks. The logic in our business layer provides for a unique approach to handling 
user specifications by creating dynamic dashboards. We highlighted the success and 
the challenges faced during the six months of deployment of our system in a residence 
which acts as proof of concept for our main objectives. It also opened the possibility of 
future developments and improvements. We target in our future work to focus more on 
the quality-of-service (QoS) nature of the platform and to have a distributed architecture 
to enhance our scalability aspect. 
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Abstract. In the literature, a wide number of annotation systems in the e-health 
sector have been implemented. These systems are distinguished by a number of 
aspects. In fact, each of these systems is based on a different paradigm, resulting in 
a jumbled and confused vision. The purpose of this study is to categorize medical 
annotation systems in order to provide a standardized overview. To accomplish 
this, we combed through twenty years' worth of scientific literature on annotation 
systems. Then, we utilized five filters to determine which systems would proceed 
to the classification phase. The following filters have been chosen: accessible, 
free, web-based or stand-alone, easily installable, functional, availability of docu- 
mentation. The classification step is performed on systems that evaluate “true” for 
all of these filters. This classification is based on three modules: the publication 
module, the general information module and the functional module. This research 
gave us the chance to draw attention to the issues that healthcare professionals 
may face when using these systems in their regular work. 


Keywords: Annotation - System - e-health - Healthcare professional - Patient - 
Interoperability - Partnership 


1 Introduction 


Annotative activity had previously occurred in medieval Latin society. The examination 
of palimpsests discovered throughout history shows that, at that time, many people 
annotated the same manuscript, which resulted in the accumulation of several layers of 
annotations. So, we can say that annotation was a necessary and crucial activity that 
allowed messages to be passed down to future generations. 

Since the invention of printing, each person has had their own annotation support 
where they can write private notes. In this way, only the owner of the annotated support 
has been able to read the annotation [1]. 

Annotative activity is now widely used in almost every aspect of life. In this regard, 
consider the case of a student who highlights essential passages in a text and makes 
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notes in the margin. Also, we can cite the example of a teacher who deciphers a number 
next to a word and then consults the bottom of the page to discover the associated note. 
Annotation types and functions differ widely, and they are classified depending on a lot 
of factors, such as the annotator’s type, the goal, the spatiotemporal frame, and so on 
[2]. 

Traditionally, annotation is done on material supports. The researchers were inter- 
ested in this approach; therefore, they took the required measures to help it evolve by 
implementing computer systems to manipulate annotations. These systems offer new 
functionalities that facilitate the tasks of the annotator. Such as: storing the annota- 
tions in files or databases separate from the original document, combining annotated 
documents, exchanging data, etc. [3]. 

When it comes to the medical field, annotation is a critical skill. Therefore, health- 
care practitioners annotate medical records that are both written on paper and stored 
electronically. The purpose of this research is to offer some insight into the annotation 
systems used in the medical field. Therefore, we will try to obtain a standardized and 
clear overview of the various medical annotation systems that have been developed in 
the literature. To achieve our goal, we will identify a strategy that brings together the 
filtration and classification of these systems. In the literature there is little research on 
the classification of medical annotation systems. However, no work has been done to 
implement a filtration strategy for these systems. Our method enables health profession- 
als to select the appropriate annotation system for their specialties while also ensuring 
that the system is operational and available. 

This strategy has the following form: To begin, we will filter the annotation systems 
that were chosen throughout the search phase. This filtering is based on five filters: {F1: 
accessible, F2: web-based or quick installation, F3: functional, F4: free, F5: availability 
of documentation}. To be elected for the phase of classification, an annotation system 
must have the value ‘true’ for all filters. The phase of classification enables the elected 
systems to be classified according to three classification modules: publication module, 
general information module and functional module. 

The following sections comprise this article: 

The second section is labeled “Annotation Systems Filtering”. This section includes 
a strategy for filtering annotation systems. This part will aid us in weeding out systems 
that do not meet our needs. 

The third section, “Annotation Systems Classification” will classify annotation 
systems using three classification modules. 

In the fourth section, "Observations", we will discuss our observations regarding the 
study of these systems. 

Finally, we will conclude our paper and give various viewpoints to round out our 
study. 


2 Annotation Systems Filtering 


Due to the important number of annotation systems, we discovered, we omitted plu- 
gins and standalone software that require the user much effort to install them locally. 
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Indeed, our main attention should be the annotation activity., avoiding all installation 
and configuration complications. 

An annotation system must be accessible, i.e., the scientific publication should 
contain a functional link that allows access to the tool. 

It must also be functional, in the sense that he completes the duties that have been 
given to him appropriately. 

Elected systems must be also free [4]. 

An annotation system must have a documentation in order to study its internal 
structure. 

The following is a list of the election filters that we have fixed: 


F1: accessible 

F2: free of charge 

F3: web based or standalone easily installable 
F4: functional 

F5: documentation availability 


A system must have the value’ true’ for all filters in order to be elected. Elected 
systems will be thoroughly investigated in the classification stage. 


Election Decision — (F1 and F2 and F3 and F4 and F5] 


The following figure (Fig. 1) shows the process of filtering annotation systems. 


selected 
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T 
elected " 
possem | 


MRejected " 


E system 2 


Fig. 1. Filtering process 


3 Annotation Systems Classification 


3.1 Publication Module 


We cannot examine an annotation system in isolation from the facts surrounding its 
publishing. This module describes elements of both the tools and its scientific publica- 
tion. These are crucial aspects to evaluate the tool’s originality. The following are the 
publication criteria that we have selected [4, 5]. 


Paper title 


Publication year 
Type of the paper 
Publication medium 
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The following table (Table 1) summarizes the publication criteria for each paper. 


Table 1. Publication module 


System’s name 


SIFRBiopotal [6] 


Paper title 


SIFR annotator: 
ontology-based semantic 
annotation of French 
biomedical text and 
clinical notes 


Publication year 


2018 


Type of the paper 


Journal paper 


Publication medium 


BMC Bioinformatics 


3DBIONOTES [7] 


3DBIONOTES v2.0: a 
web server for the 
automatic annotation of 
macromolecular structures 


2017 


Journal paper 


Bioinformatics 


RILcontour [8] 


RIL-Contour: A Medical 
Imaging Dataset 
Annotation Tool for and 
with Deep Learning 


2019 


Journal paper 


Journal of Digital 
imaging 


RNAmod [9] 


RNAmod: an integrated 
system for the annotation 
of mRNA modifications 


2019 


Journal paper 


Nucleic Acids Research 


VarAFT [10] 


VarAFT: a variant 
annotation and filtration 
system for human next 
generation sequencing 
Data 


2018 


Journal paper 


Nucleic Acids Research 


VADR [11] 


VADR: validation and 
annotation of virus 
sequence submissions to 
GenBank 


2020 


Journal paper 


BMC Bioinformatics 


ODMSummary [12] 


ODMSummary: A Tool for 
Automatic Structured 
Comparison of Multiple 
Medical Forms Based on 
Semantic Annotation with 
the Unified Medical 
Language System 


2016 


Journal paper 


PLOS ONE 


Heideltime [13] 


HeidelTime Standalone 
Manual Version 2.1 


2015 


Workshop paper 


Proceedings of the 5th 
international workshop 
on semantic evaluation 


Marky [14] 


Marky: A tool supporting 
annotation consistency in 
multi-user and iterative 
document annotation 
projects 


2015 


Journal paper 


Computer methods and 
programs in 
BIOMEDECINE 


(continued) 
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Table 1. (continued) 


System’s name 


Biocat [15] 


Paper title 


BIOCAT: a pattern 
recognition platform for 
customizable biological 
image classification and 
annotation 


Publication year 


2013 


Type of the paper 


Journal paper 


Publication medium 


BMC Bioinformatics 


GATE Teamware [16] GATE Teamware: a web- | 2013 Journal paper Language Resources & 
based, collaborative text Evaluation 
annotation framework 
BioQAator [17] BioQRator: a web-based 2013 Workshop paper Proceedings of the 
interactive biomedical Fourth BioCreative 
literature curating system Challenge Evaluation 
Workshop 
Anafora [18] Anafora: A Web-based 2013 Conference paper | Proceedings of the 
General Purpose conference Association 
Annotation Tool for Computational 
Linguistics 
Gap-mind [19] GapMind: Automated 2020 Journal paper Msystems 
Annotation of Amino Acid 
Biosynthesis 
QUICK Annotator [20] | Quick annotator: an 2021 Journal paper The journal of pathology 
open-source digital 
pathology based rapid 
image annotation tool 
imageJ2 [21] ImageJ2: ImageJ for the 2017 Journal paper BMC Bioinformatics 
next generation of 
scientific image data 
Annotation web [22] Annotation web-An open- | 2021 Journal paper IEEE International 
source web-based Ultrasonics symposium 
annotation tool for (IUS) 
ultrasound images 
CARD [23] A long journey to short 2017 Journal paper Journal of the Amecican 
abbreviations: developing Medical Informatics 
an open-source framework Association 
for clinical abbreviation 
recognition and 
disambiguation (CARD) 
LesionMap [24] LesionMap: A Method and | 2020 Journal paper JMIR Dermatology 
Tool for the Semantic 
Annotation of 
Dermatological Lesions 
for Documentation and 
Machine Learning 
VIA [25] The VIA Annotation 2019 Conference paper | Proceedings of the 27th 
Software for Images, ACM international 
Audio and Video conference on 
multimedia 
OpenMRS [26] Relationship-Based Access | 2015 Conference paper | Proceedings of the 20th 
Control for an Open- ACM symposium on 


Source Medical Records 
System 


Access control Models 
and technologies 
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3.20 General Information Module 


[3, 27] The annotation activity begins by selecting the anchor and the shape of the 
annotation from the software toolbar. Then, the annotation must be attached to a well- 
defined target in order to satisfy all of the annotation’s requirements. This annotation 
activity can be classified into three broad types. 


e Manual format: this format gives complete responsibility to the user for the annotat- 
ing process. He begins by selecting the annotation’s shape, followed by the anchor and 
lastly the annotation itself. This is comparable to how annotating on paper is handled. 


e Automatic format: the machine is programmed to carry out the entire annotation 
process without human intervention. 

e Semi-automatic format: in this situation, the user initiates the process. The algo- 
rithm eventually learns and understands how the user annotates. It then suggests 
automated annotations based on an annotation model developed with rules in devel- 
opment. When no adjustments are made and the suggested rules are fully accepted, 
human intervention is cancelled and the process becomes fully automated. 

Annotation can be classified into two types. 

e Cognitive annotation: this type of annotation has a visible form on the document. 
Because it is employed by human agents, comprehending it requires cognitive and 
mental effort. 

e Computational annotation: sometimes referred to as ‘meta-data’. The annotation 
is treated and manipulated by software agents. 

The annotation has two types of structures: 

e Unstructured annotation: in this situation, each annotator annotates in accordance 
with his requirements. 

e Structured annotation: the annotation can be based on well-defined models and 
forms; in most cases, this type of annotation is carried out as a result of agreements 
reached amongst a group of people working together. 

The information in the table below (Table 2) highlights a classification of 
annotation systems based on the following characteristics: 

e Link allowing access to the tool 

e Concerned medical field: (biology, radiology, doctor, biochemistry, all healthcare 
professions } 

e Annotation type: (cognitive, computational] 

e Annotation activity type: (automatic, semi-automatic, manual] 

e Annotation structure: (structured, unstructured} 
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Table 2. General information module 


System’s name Link allowing access to the Annotation Annotation Annotation Medical 
tool type activity type structure field 
© Ò z o g S B S 
O IS *|^os|2|a 2 
8 
SIFRBioportal ttp://bioportal.lirmm.fr/ iology 
3DBIONOTES 3dbionotes-ws.cnb.csic.es/ws Biology, 
biochemistry 
RILcontour ttps://gitlab.com/Philbrick/ril radiology 
contour 
RNAmod https://bioinformatics.sc.cn/R iology 
NAmod 
VarAFT ttps://varaft.ru iology 
VADR ttps://github.com/ncbi/vadr iology 
ODMSummary https://odmtoolbox.uni- doctor 
muenster.de/summary/summary.h 
tml 
heideltime ttps://github.com/HeidelTim iology 
e/heideltime 
Marky http://sing.ei.uvigo.es/marky radiology 
BioCat ttp://faculty.cs.niu.edu/-zhou iology 
/ tool/biocat/ 
GATE Teamware ttps://gatecloud.net all 
BioQAator http://www.bioqrator.org/ all 
Anafora ttps://github.com/weitechen/ all 
anafora 
Gap-mind https://papers.genomics.Ibl.go biology 
v/cgi-bin/gapview.cgi 
Quick annotator ttps://github.com/choosehap radiology 
py/QuickAnnotator 
imageJ2 ttps://imagej.net/ al 
Annotation web https://github.com/smistad/an radiology 
notationweb/ 
CARD https://sbmi.uth.edu/ccb/resou al 
rces/abbreviation. htm 
LesionMap https:/ doctor 
/github.com/dermatologist/lesion- 
mapper 
VIA ttp://www.robots.ox.ac.uk/~v al 
gg/software/via 
OpenMRS http://openmrs.org/ al 


3.3 Functional Module 


Each annotation system has a particular variety of features [3, 28]. 


Sharing Features: 


e Annotation export: the annotator wishes to send all or a portion of the annotations 
that have been written on a document. 

e Annotation import: the user can receive annotations. This feature enables him to add 
new annotations to a document as if it had been annotated by two different annotators. 

e Sending a message: A healthcare professional sends a message with an annotated 
document or record. So that healthcare professionals can converse asynchronously. 
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Memorization Features: 


e Annotation creation: there are two methods for creating annotations: 

e Annotation modification: the annotator has the ability to change all of the 
annotation’s parameters (shape, color, content etc.). 

e Delete of annotation: the annotation can be removed without being archived. 

e Annotation saving: an annotation can be saved in a variety of formats (text, XML, 
etc.) 

e Reading and browsing the document: access to the document should be granted 
to the user. If that’s the case, the reader opens the annotation system and chooses 
an existing document. He can use the mouse, keyboard arrows, and the elevator to 
navigate to the next and previous pages, as well as the beginning on finish of each 
page. 

e Visualization of the annotation in the document: the annotations are scattered 
throughout the main document. 


Reuse Features: 


e Filtering: the reader is looking for one or more annotations that meet certain 
requirements. 


e Manual: Depending on the user’s preferences, the health care provider can choose 
to see only a subset of annotations. 

e Automatic: Only annotations that have been granted permission to be seen by the 
healthcare professional are visible to him. 


e Visualization of the annotation outside the document: annotations are displayed 
in a different location than the primary document. 

e Segmentation: image segmentation is a type of image processing that seeks to group 
pixels together based on established criteria. 

e Sorting: the list of displayed annotations is organized by sorting annotations based 
on their attributes. 

e Merging of annotated documents: this feature allows the user to create a report 
that includes annotated documents. Based on the annotation, the merging produces a 
summary of the patient’s condition. This process enables experts to share documents. 

e Comparison of annotations: this comparison seeks to determine whether or not two 
given annotations have the same meaning. 

e Redefinition of an annotation: the practitioner manually traces any annotation, and 
then the machine automatically intervenes to retrace it. 

e Recommendation: this feature allows the user to provide suggestions for possible 
annotations. 
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e Localization of the annotation and calculation of the area of the annotated zone: 
this functionality allows the user to specify the coordinates of the anomalous compo- 
nent (sick) and determine its interface by locating the annotation and calculating the 
area of the annotated zone. 

e Annotation search: looking for an annotation based on a number of parameters. 

e Standardize annotations: transform annotations into a standard format. 


The following table (Table 3) includes a classification of annotation systems 
according to functionalities. 


Table 3. Functional module 


Sharing Memorization features Reuse features 
features 


lization in the 


Import 
Creation 
Modification 
Filtering 

Sorting 

Merging 
Comparison 
Search 
Standardize 
Recommendation 


Delete 


SIFRBioportal 
3DBIONOTES 


RlILcontour 


RNAmod 


VarAFT 


VADR 
ODMSummary 


heideltime 


Marky 


BioCat 
GATE 


BioQAator 


Anafora 


Gap-mind 


Quick annotator 


imagel2 


‘Annotation web 


CARD 


LesionMap 
VIA 


OpenMRS 


4 Observations 


This survey allowed us to address the research question posed in the introduction of the 
article, which serves to obtain a uniform and clear overview of the different exciting 
annotation systems in the literature. 

By finding the answer to the research question, we now have a better understanding 
of the issues that the healthcare providers may run into when utilizing these tools. 


Interoperability Problem: each of the systems examined above is based on a set of 
standards and formats unique to that system. This results in an issue with the integration 
of heterogeneous programs, and as a result, there is a communication blockage. there 
is no longer any exchange of data between the apps. A health professional cannot send 
or receive data from his colleagues. This creates a problem for both patients and health 
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professionals because it restricts their mobility. Additionally, the patient is unable to 
communicate his personal data to his doctor, resulting in data loss. To circumvent this 
issue, harmonizing annotation systems’ models and standards is required. 


Problem of Patient Integration: because the patient is unable to access the content of 
the annotation systems, he can no longer communicate his ideas in a meaningful way. 
In addition, he finds himself in a state of uncertainty as to his understanding of the 
notes that have been placed in his medical file. Indeed, he does not master the scientific 
language and medical terminology provided by his doctor, which leads him to ignore 
his current state of health, his medications, and his entire treatment protocol, which can 
sometimes make the recovery more difficult. In this way, the annotation is unable to 
promote collaboration between different stakeholders. But, on the contrary, it leads to a 
breakdown in communication between health professionals and patients. 


No Development of a Partnership Cycle: with the integration of this cycle, the anno- 
tation will ensure sharing and communication between all stakeholders. This must be 
done to reach a consensus on a decision. 


Problem of Deciphering the Structure of Annotation Systems: several of the sys- 
tems reviewed are open sources, which allows for the examination of their documenta- 
tion and source code in order to gain a better understanding of their structure. However, 
decoding the structures of other systems is difficult, if not impossible, because we no 
longer have access to their source codes. 


Hardware restrictions: the only data entering materials are a keyboard and a mouse. 
Improved data entry assistance devices, such as the stylus, are required. The stylus is 
a tool that can make healthcare professionals’ annotation activities easier and more 
versatile. 


5 Conclusion 


Towards the end of this article, we will briefly say that we have proposed a strategy for 
the filtering and classification of annotation systems in the medical profession. This has 
allowed us to draw attention to the problems healthcare professionals may encounter 
when using these systems in their daily activities. 

As part of our future study, we want to focus on establishing an annotation modeling 
standard that allows both healthcare professionals and patients to annotate electronic 
medical records. 
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Abstract. With the rise of the digital age, life has become a lot easier for 
the vast majority of the population. However, the ever-increasing elderly 
population has suffered, especially in countries like Pakistan, where lim- 
ited accessibility to technology, often due to language barriers, hinders 
elderly from reaping technological benefits. In this paper, an Urdu vir- 
tual assistant application is proposed which provides an intuitive and 
empathetic platform for the elderly in Pakistan that helps them perform 
essential tasks such as reminding them of their medications, organising 
their work, getting daily news highlights, and connecting them with their 
loved ones. It also provides entertainment in the form of user-specified 
video playlists or by positively engaging them in conversations on various 
topics. 


Keywords: Elderly care - Virtual assistant - Conversational agent - 
Urdu natural language understanding (NLU) 


1 Introduction 


According to the World Health Organization (WHO), life expectancy is increas- 
ing due to advancement in science, strong economies, and healthy behaviours 
[10]. Due to this increased life expectancy, most people are expected to live 
beyond their sixties, allowing them to contribute to society as mentors, innova- 
tors, and in a variety of other ways. The latest census of Pakistan states that 
the current population of people over the age of 60 (old-age) is estimated to be 
5.5496 [2] and this number is set to increase. However, due to the increasing nor- 
malisation of a nuclear family structure [20] in Pakistani society and the hectic 
lifestyles of modern-day individuals, oftentimes there are not enough people to 
take care of older family members. As a result, the elderly might end up iso- 
lated and confined to their homes. This can negatively impact their emotional 
and physical well-being. Consequently, there is a growing demand for technology 
that supports the elderly at home [11] and can assist them in their daily tasks 
helping them stay connected to their loved ones. 
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To address the aforementioned problems, a solution called “Saathi” is pro- 
posed. Saathi intends to offer the elderly a supportive environment by allowing 
them to use an Urdu mobile virtual assistant to perform everyday tasks such as 
setting medicine and appointment reminders, calling their loved ones, navigat- 
ing through different apps in their phones using visual icons or voice commands 
in Urdu to improve their everyday life. In addition to that, it provides them 
with entertainment, prompts them to take their meals on time, and generates 
empathetic conversation or actions accordingly. While an abundance of such 
applications exists in languages like English, there are currently no assistive 
applications for the elderly that support the Urdu language. Saathi acts as a 
bridge that allows the elderly to use their smartphone by making it more acces- 
sible through a voice-enabled Urdu virtual assistant as well as a consolidated 
and user-friendly user interface. 

The rest of the paper is organized as follows. In Sect.2, the existing work 
is reviewed in the domain of Socially Assistive Robots (SAR) and virtual assis- 
tants (VA) for elderly. Section 3 explains the proposed solution which includes 
the overall architecture and features of the application. Section 4 presents the 
application prototype and the results related to testing performed on the conver- 
sational agent. Section 5 discusses the obtained results and mentions the future 
scope of the project and Sect. 6 concludes the paper. 


2 Related Work 


Much research and development has been done in the domain of Socially Assis- 
tive Robotics (SAR) to develop companion robots to assist older people in a vari- 
ety of tasks such as monitoring and promoting physical health in the elderly at 
home [3,11], helping them by autonomously navigating the user's home, remind- 
ing them of their medications, and providing entertainment [1]. Other than that, 
technologies such as Internet of Things (IoT) and wearable technologies could 
offer promising solutions for elderly care. Wearable devices are a part of IoT 
systems which can be worn by the people for monitoring the physical activi- 
ties and physiological data [19]. These devices are embedded with sensors and 
algorithms that allow them to track, analyse and guide users’ behaviour [17], 
vital signs or movement. Wearable tech has the potential to assist with several 
scenarios in elderly healthcare and can improve the quality of life for the elderly 
and allow them to maintain their independent lifestyle [19]. However, one of the 
major issues with SAR systems and wearable technology is the lack of social 
component and interaction, which hinders their acceptance among older people. 
Another problem with these assistive robots is their significant cost, which is not 
within everyone's budget, especially in low income countries like Pakistan. 

Like other technological devices, virtual assistants have also become an ele- 
ment of great help to integrate and keep people active, in addition to facilitating 
the different everyday tasks that need to be performed [13]. A virtual assistant 
is an application that recognises voice instructions and does tasks on the user's 
behalf. 
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There has been significant work done to develop user-centered virtual assis- 
tants that feature different functionalities to assist elderly people. Researchers 
have found that spoken language seems to be the most preferred mode of interac- 
tion for elderly people, they prefer to interact with virtual assistants in a simple, 
hassle free way [18]. Beskow et al. [6] designed a user-centric virtual assistant 
that could be used to set medicine reminders after surgery and had a virtual 
character with whom users could communicate and set reminders, among other 
things. Lesin [16] developed a virtual assistant solely for mobile platforms, which 
aimed to assist diabetic patients with their daily routines. In addition to provid- 
ing reminders, it also possessed the option to reach out to doctors/medical staff. 
Bickmore et al. [7] developed a virtual assistant to foster fitness and exercises 
in elderly people. Mival et al. [12] developed a virtual assistant that resembled 
a dog so that people may have virtual pets and communicate with them. Klein 
[15] created a virtual assistant that could aid a user in managing stress and 
anxiety by offering a variety of solutions for reducing stress and anxiety. Kasap 
et al. [14] created a virtual assistant that could change its facial expressions 
depending on how the user was feeling. Yasuda [21] created a virtual assistant 
with a human-like aspect to encourage user attention in a purely conversational 
medium. 

'The aforementioned works are in English and hence would not be helpful in 
the local context of Pakistan as majority of the population does not converse in 
English. Little work has been done in the field of virtual assistants in Urdu. In 
2019, C-Square collaborated with Genesys to launch Pakistan's first AI enabled 
Urdu voice recognition bot named RUBA (Real Urdu Bot Automation) [4] at 
the Smart CX conference. RUBA stands for Real Urdu Bot Automation and is 
a Siri-styled virtual voice-enabled personal assistant. The bot can converse with 
the user in Urdu and react in Urdu while performing simple activities like as 
checking the amount of the user's bank account, sending text messages, and so 
on [4]. However, the domain of this app is very limited and does not focus on 
solving problems specific to any particular group in the Pakistani society. 

'There are a few other mobile applications available such as Carer, Elderly 
Care and BaldPhone which are designed for elderly people, however all of these 
apps are solely in English and not voice activated hence they would be of little 
use to the majority of Pakistani elderly individuals who do not communicate in 
English. 


3 Proposed Solution 


Saathi is an Urdu virtual companion application for elderly people in Pakistan. 
It facilitates them in their daily tasks and provides them with a supportive envi- 
ronment to keep them engaged and entertained through technology. Saathi is 
built using Flutter for the frontend, Firebase for the backend and the conversa- 
tional agent has been designed using the RASA Framework. Figure 1 represents 
a high-level architecture of Saathi. 
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3.1 Conversational Agent 


The Urdu conversational agent is based on the RASA framework. RASA is 
a Python based, machine learning framework for conversational AI [8]. Since 
RASA is an open-source conversational AI framework, its code is readily avail- 
able to use for free. It is an accessible, flexible, robust, and transparent frame- 
work. A benefit that RASA provides is that it follows a modular, extensible, 
micro-services architecture that fits well in any typical software development 
scenario. It is easy to integrate and customize to fit the needs of our application. 
Most importantly, RASA allows to develop conversational agents that leverage 
the power of NLP to determine user intents and mimic human-like conversation, 
as compared to chatbots or conversational agents with hardcoded logic and no 
capacity of learning. 

Saathi uses RASA to engage the elderly in chit-chat about subjects that are 
most common among Pakistan’s senior citizens. These topics of interest could 
include food-related discussions, asking them to describe their past experiences, 
or conversing with them about their favourite activities. Along with daily con- 
versations, the conversational agent can be used to respond to queries about the 
weather, news, and time among other things. 

RASA framework breaks down the processing of user queries and returning of 
responses into two main components, namely, Natural Language Understanding 
(NLU) and Core Dialogue Management [8]. Figure 2 shows the complete data- 
pipeline for the conversational agent module. 


Natural Language Understanding (NLU) Component: The NLU com- 
ponent of the RASA framework is further divided into two components: 


1. Natural Language Processing (NLP) Component: The raw Urdu text 
is first passed through the Urdu Natural Language Processing pipeline built 
using the SpaCy library. The Urdu NLP pipeline consists of a tokenizer, 
a parts-of-speech tagger, a parser and a named entity recognition module. 
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Fig. 2. Data pipeline for RASA framework 


The tokenizer breaks the input sentence into smaller pieces called tokens. 
These tokens are tagged and categorized in correspondence with a particular 
Urdu part of speech by the tagger. The parser then extracts the syntactic 
structure of the input text by analyzing the words based on the grammar 
of the language. Lastly, the named entity recognition module identifies and 
classifies the named entities in the text. These named entities refer to the 
proper nouns found in the text. 

2. Intent Classifier and Entity Extractor: The processed query from the 
NLP component is passed on to the intent classifier and the entity extrac- 
tor module. Entities in RASA are structured pieces of information inside a 
user message [8]. The module maps the user query to a pre-defined intent 
and extracts important entities from it. RASA uses DIET (Dual Intent and 
Entity Transformer) architecture as part of its NLU component. DIET is a 
multi-task transformer architecture that handles both intent classification and 
entity recognition together [9]. The DIET architecture comes with the abil- 
ity to plug-and-play various pre-trained embeddings and support for custom 
components and pipelines to use any other ML model. 


Core Dialogue Management Component: The extracted intents and enti- 
ties from the NLU component are then passed on to the second component of 
the framework called RASA core dialogue management. RASA core uses a clas- 
sifier as a response selector. The response selector finds and outputs the best 
response to the user input query. In the case where the RASA module cannot 
map a query to a pre-defined intent, a fallback policy is activated by the RASA 
module which sends a fallback response to user and asks the user to rephrase 
their query. 


Custom Actions: Sometimes, the queries by users might require the execution 
of custom code to process the response required by the user. Such queries are 
handled by the custom actions module in RASA. RASA's custom actions module 
allows running custom code which can be used to perform database queries, make 
API calls, etc. Saathi uses custom actions to add events to reminders, open 
mobile applications, dial calls, send messages, etc. Following are a few custom 
actions that the user can perform using Saathi conversational agent: 


1. Feeling connected: Saathi allows the elderly to open any application that 
might be installed in their smartphones through Urdu voice commands. It 
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also helps them to dial a call or send a message to their loved ones by using 
either their name or phone number. These actions are performed by using 
an android intents plugin supported by Flutter. Such features help keep the 
elderly feel connected and give them a sense of independence. 

2. Reminders: While the elderly are physically capable of performing most 
everyday tasks, they frequently forget to do so. Therefore Saathi can remind 
them of their medicines, meals, and any events they choose to be reminded 
of. They can do this through voice commands which triggers a custom action 
that takes the date, time and reminder description as entities and adds a 
reminder to the user’s schedule. Other than user-defined reminders, there are 
reminders like reminding the elderly to charge their phone when the battery 
is running out. 

3. Entertainment: According to research, the elderly’s entertainment needs are 
equally vital for their well-being and joyous life [5]. Saathi several features 
for providing daily entertainment to the elderly. At the time of sign up, they 
can specify a playlist they prefer to listen to and whenever they want to play 
it, they can just ask the conversational agent to do so. Moreover, they can 
also get the latest news, ask about the weather of a particular city or inquire 
current time. 


3.2 Empathetic Responses 


It is important to emphasize that all intents and responses in the conversational 
agent have been pre-defined in Urdu. This means that the scope of the applica- 
tion has been set up during the development phase. The responses and interac- 
tions are tailored in such a way that they are as empathetic and human-like as 
possible. Firstly there are intents, which are used to categorize user messages. 
For example, for an intent called “greet” all the different kinds of greetings in 
Urdu are defined. Whereas, responses are what the chatbot sends to the user. 
After defining all the anticipated messages from the user and the replies from 
the chatbot, multiple intents and responses are tied together to form stories, 
which act as templates for possible conversations. These templates are provided 
as training data for the chatbot’s dialogue management model [8] which in turn 
helps generalize the model to deal with unseen conversation paths. In Fig. 3, the 
story shows a happy path where the user might say hello and the chatbot greets 
them in return. It then asks them about their mood and depending on the mood, 
different story paths are chosen. In this particular example, the user’s mood is 
great and so the chatbot replies with an appropriate response. 

The RASA conversational agent requires datasets in the form of intents and 
responses. For our application, we have defined 28 intents in total for now, cov- 
ering a range of queries, concerns and actions that the elderly might require. 
However, this set of intents can further be extended depending on the future 
needs of our application. Each of these intents have 10-15 Urdu examples on 
which our model has been trained. These example sentences have been crowd- 
sourced from native Urdu speakers to train the model to best identify each 
requisite intent. Each intent is associated with a response or custom action that 
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Fig. 3. RASA file format 


is sent as output to a triggered intent to help the elderly. RASA allows making 
the conversational agent’s replies more interesting if multiple response variations 
are provided to choose from for a given response name. Where possible, we have 
added variability in responses by providing multiple possible response utterances 
mapping to specific intents. RASA then selects one of the provided utterances 
at random and sends it to the user. 

In contrast to question-answer bots, Saathi is categorized as a companion or a 
conversational agent. Therefore, special care is taken in appropriately responding 
to different scenarios. One such example is when the elderly user mentions that 
they are feeling sad or anxious, an empathetic response is sent, which asks them 
if anything can be done to help improve their mood. In some cases, a YouTube 
video of narration of an Urdu short story is also sent to cheer them up. 


3.3 User Interface 


To use Saathi, the user first needs to sign up for the application and fill in the 
required information such as their email, password, medicine timings, and pref- 
erences. These preferences can be related to their meal, medicine or appointment 
reminders, specifying a Youtube playlist they like to watch, etc. The registration 
process also requires the user to provide information of at least one caretaker, so 
that they can be alerted in cases of emergency such as when the elderly might be 
feeling unwell. After signing up, the elderly can interact with the application and 
use features such as the visual assistant, which allows them to navigate through 
their phones easily, the scheduler, where they can see their reminders and events, 
and the virtual assistant which can be used by their voice in the Urdu language. 

To use the virtual assistant, the user can give their queries in the form of 
speech or voice commands in Urdu. The Urdu voice data is first converted to 
text using the speech to text feature of the underlying mobile platform and then 
passed on to the conversational agent module in the form of a text query. On 
receiving the text response from the conversational agent, it is converted to voice 
using the text to speech feature and sent to the user. 

Saathi relieves the caretakers - people in charge of looking after the elderly - 
the burden of constantly being worried about the well-being of their elderly loved 
ones. This is done by keeping them up to date on the elderly person’s medicine 
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and meal intake, the status of their appointments and other such events. To 
update the caretaker, a daily log of above-mentioned activities is maintained 
and shared with them on WhatsApp using the Twilio API. 


4 Prototype and Results 


This section presents the application prototype to validate the proposed solution 
and the results obtained while testing the conversational agent. 


4.1 Application Prototype 


The mobile application is installed on the elderly’s smartphone. It is used to 
communicate with the conversational agent which is deployed on the cloud. 
Through the application, the elderly can navigate to different screens and choose 
the activities that they are interested in performing. The application allows them 
to do the following tasks: 


1. Registration and Login Screens: Used to register their information, pref- 
erences and caretaker information as shown in Fig. 4. 

2. Visual Screen: A screen which displays all the features consolidated in a 
single place for easy navigation as shown in Fig. 5a. 

3. Conversational Agent Screen: Used to chat with the conversational agent 
in order to perform various tasks or engage in chit chat as shown in Fig. 5b. 

4. Schedule Screen: Used to set, update and delete reminders related to meal, 
medicine, medical appointments and other such events as shown in Fig. 5c. 


4.2 Prototype Validation 


In order to test the application, unit and integration tests were written for dif- 
ferent components and modules of the application. Usability testing was also 
performed to evaluate the accessibility and ease of use of the application from 
the user’s perspective. The testing was carried out on 10 users over the age of 
60. One of the feedbacks received from the users was that they faced difficulty 
in navigating to different screens of the application. Another issue pointed out 
was the limitation in the type of conversations with the conversational agent. 

After getting the feedback, the user interface was made more intuitive and 
easy to use for the elderly by increasing the font, icon and image sizes. To provide 
ease in navigation, we provided a visual screen where all the basic features are 
consolidated on a single screen. For the conversational agent, more data was 
added around different topics such as checking elderly mood, discussing their 
interests, likes and dislike to keep the elderly engaged. 
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4.3 Results for Conversational Agent 


Since the conversational agent will be directly interacting with elderly people it 
must be as accurate and well-performing as possible. To do this, the model was 
tested through the RASA test module. The cross-validation method was used 
to test the chatbot, where the dataset is split into k number of groups, each of 
equal size, in this case, k = 5. One of these k splits is then chosen for testing and 
the rest of the splits are used for training the chatbot. This process is repeated k 
number of times until each split is used for testing. Finally, a weighted average of 
all k iterations is taken to validate the model’s performance. Figure 6 shows the 
confusion matrix of intents with the cross-validation approach. It shows that even 
though the diagonal values are high, there are several misclassifications of the 
intents. The ratio of misclassifications to correct classifications is 0.247. While 
the model performs well on a majority of the intents even with cross-validation, 
there is still room for improvement. Due to these misclassifications, a threshold 
of 0.7 is set as the confidence level of an intent classification. This implies that 
if the confidence score of any intent goes below the defined threshold value, a 
fallback policy is triggered and the user is asked to rephrase the sentence. The 
overall goal is to ensure that no inconvenience or discomfort is caused to the 
user by any incorrect or hurtful responses. 


5 Discussion 


From the results, it is apparent that the conversations are not strictly domain- 
specific and can instead be more open-ended. As the conversations are open- 


82 A. Kumar et al. 


kJ 
Visual Assistant € . Scheduler 
alo LS gU < March 2022 > 
Q Mon Tue Wed Thu Fri Sat Sun 
em uua 29 5 03 S ali 1 2 3 4 5 6 
Talk to your Saathi 
u$ Sb cra ttle cul p 7 8 9 10 12:913 
f] 14 15 16 az: 18 19 20 
\ DM 21 22 23 24 25 26 27 
Call Message eS) JUS bal 
oda JIS Tga ausi 28 29 Á 30 3 
Fri, March 11 
m | vou Gils 
=... 
HH ) LÍ a uil ail Sly, oly 1 casual ALS ule 
Daily Schedule Add Reminder os 2) Jonas sls IS eT al. Ulas 


asas ys uns Jala 3b ab Blas oly 


(a) Visual Screen (b) Chat Screen (c) Schedule Screen 


Fig. 5. Application prototype 


ended, it is difficult to account for all possible conversational intents and their 
relevant replies. This is where a need for an Urdu conversation generation model 
arises. 

However, to our knowledge, no such model exists to perform this task in the 
Urdu language. Hence, this area can potentially be explored further. 

Urdu, being Pakistan's national language is understood by almost everyone 
irrespective of their ethnic identity. For a language so widely spoken, there need 
to be technological advancements so applications like Saathi can be developed 
with ease. The features that makeup Saathi are usually only provided by applica- 
tions that do not support the Urdu language and most elderly people in Pakistan 
are not fluent English speakers. This huge gap makes technology inaccessible for 
around 15 million [2] elderly people of Pakistan. Additionally, Urdu being a 
resource-poor language with respect to the NLP resources currently available 
means that it is very difficult for researchers and developers to effectively solve 
Urdu NLP related problems like Saathi. 
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6 Conclusion 


There are many virtual assistants that exist in English and other commonly 
used languages specifically designed for the elderly population. However, no 
such application currently exists in the Urdu language as per our knowledge. 
This paper presented a one of its kind Urdu language based virtual assistant 
named “Saathi”, in the form of a mobile application. It can help the Urdu- 
speaking elderly individuals navigate through their smartphones with ease and 
provide them with a supportive environment to stay entertained and engaged in 
daily activities. Saathi serves as a companion to the elderly and helps them stay 
connected with their loved ones. 
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Abstract. This study evaluated a smart technology intervention in the home as 
a support for individuals with severe mental illness. This study recruited 13 par- 
ticipants in a variety of community-based homes. Participants were offered a 
smartphone, a touchscreen monitor and health devices such as smartwatches, 
weigh-scales, and automated medication dispensers. Data was exported to the 
Lawson Integrated DataBase for care providers to monitor/track. Interviews with 
participants and focus groups with participants and care providers were conducted 
at baseline, 6-months and 12-months, and survey instruments were used to col- 
lect quantitative data about different dimensions of health and social determinants. 
Descriptive statistics from these outcome measures are presented as the sample size 
was too small for meaningful statistical inference. Qualitative analyses revealed a 
high degree of acceptability of the devices and motivation for healthy living, com- 
munication and mental health. Health Care Providers also noted improvements to 
client health. This study proves the feasibility of deploying smart technologies to 
support individuals with severe mental illness. Future scale-up would further our 
understanding of their impacts. 


Keywords: Smart technology - Smart homes - Mental health - eHealth - 
Physical health 


1 Introduction 


Smart homes to support mental illness are an appealing new concept but have not yet been 
implemented at the community level as a form of mental health care or support. Literature 
reviews have found that hospitalizations, burden on care providers, consultation and wait 
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times, and health care costs can all be reduced through health monitoring technology 
[1]. Although new technologies are rapidly developed for the consumer market, many 
are not stringently tested in a mental health care setting; leading to a lack of robust 
literature. This rapid pace of development also means that new technological advances 
become obsolete quickly over time [2]. 

Mobile technologies such as smartphones have been utilized to support mental ill- 
nesses such as bipolar disorder [3] and schizophrenia [4]. Patient beneficence and well- 
being can also be supported using smart technology by providing greater access to infor- 
mation and resources, as well as through symptom tracking and monitoring features [5]. 
A systematic review of smart technology for mental illness detailed that smartphone 
applications were the most studied, but monitoring and adherence supports were lack- 
ing significantly [6]. A more recent systematic review also revealed that research into 
technological interventions based on the Internet of Things model is still lacking, with 
mental health data scattered and segregated depending on the vendor/platform for these 
devices [7]. A smart technology intervention should require minimal input and provide 
continuous monitoring passively that does not interfere with daily activities. 

Physical activity can also be an indicator for changes to mental health and/or indica- 
tive of a crisis. Including physical activity into daily routines can be difficult, particularly 
as symptomology, motivation, experience, fatigue and poorer access to resources are 
common considerations for people with mental illness [8]. This is particularly pertinent 
during the current time with physical activity and mean peak heart rate readings decreas- 
ing significantly during COVID-19 lockdowns [9]. As such, the use of smart technology 
to maintain a healthy lifestyle is even more needed, particularly among vulnerable indi- 
viduals who may experience barriers due to their health status, lack of accessibility or 
socioeconomic status. 

Early inceptions of the current study, each lasting 12 months, attempted to provide 
supportive systems within an individual’s environment in hospital settings and tran- 
sitional hospital apartments [10]. The prototype intervention provided in the hospital 
setting was largely successful. This study set out to establish the use of smart tech- 
nology in assisting individuals with mental illness and physical comorbidities living in 
housing provided by the Canadian Mental Health Association (CMHA) and London- 
Middlesex Community Housing (LMCH). The objective of this study was to establish 
and evaluate smart home technology in the community to assist people with mood and 
psychotic disorders. We hypothesized that the smart technology would: a) increase levels 
of community integration; b) increase housing stability; c) decrease health, justice, and 
social service utilization; and d) support mental and physical health. 

The study also sought to answer the following research questions: a) What are client 
and staff experiences of smart mental health homes? b) How do clients and staff perceive 
the utility of smart technologies in the home? c) What improvements to the technologies 
do they suggest? d) What ethical issues are identified with the use of smart mental 
health homes? e) What policy issues are identified with the use of smart homes? f) What 
commercialization issues are identified by key stakeholders in relation to smart homes? 
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2 Method 


Design 

This study used a within-group, mixed-methods, repeated-measures design. Interviews 
were conducted over three assessment timepoints conducted at baseline (Time 1), 6- 
month (Time 2) and 12-month (Time 3) follow-ups. The assessments included an indi- 
vidual interview with each participant. However, due to the COVID-19 pandemic, inter- 
views and focus groups were switched to virtual and telephone formats, and focus groups 
were conducted via a one-to-one discussion instead to maintain safety as well as con- 
venience for participants. Ethical approval was obtained through Western University’s 
Research Ethics Board. 


Sample 

The research team first recruited health care providers who then referred potential par- 
ticipants for the study. Participants from a range of housing types were eligible to take 
part. The inclusion criteria for participants were as follows: 1) Must be on a caseload of a 
participating health care provider; 2) Able to understand English to the degree necessary 
to participate; 3) Living in housing provided by the CMHA or LMCH; 4) Diagnosed 
with a psychotic or major mood disorder; 5) Must be between the ages of 18-85 years 
old; 6) Able to provide informed consent. 


Intervention 

This study consisted of two interactive platforms, the Lawson Integrated Database 
(LIDB) and the Collaborative Health Record (CHR). The LIDB is an information man- 
agement platform that collates and manages health data and is protected behind the 
St.Joseph’s Health Care hospital firewall [11]. It is a web-based application that can be 
accessed by health care professionals and research staff via secure log-in. The LIDB 
connects to the clouds of health monitoring devices and is able to automatically export 
data from each cloud encrypted authentication keys and SSL connectivity. The CHR 
allows health care providers to send questionnaires (“Qnaires®”’), both standardized or 
customized, as an SMS text message, email or both to participants. The link then opens 
to a custom webpage with the questionnaire for participants to complete. The platform 
has the additional feature of videoconferencing and instant messaging. Care providers 
were able to securely log-in to the LIDB and CHR to view data from their participant's 
health monitoring devices and Qnaires® respectively. 

Screen devices offered included smartphones (Samsung J3®) and wall-mounted 
touch-screen computer monitors powered by a Raspberry Pi-3 B+® mini-computer. 
The latter were developed by the research team programmer who also designed and 
customized the interface. These devices received and responded to prompts from the 
LIDB to assist participants experiencing cognitive deficits and to facilitate self-care. 
Participants were able to “acknowledge” prompts and reminders on the touchscreen 
monitors by pushing a “Got It" button on screen which sent an automated email to 
the care providers who set that reminder. Additionally, a button which says "Please get 
in touch with me" was included, which sent a message to the participant's care team 
requesting them to schedule a meeting. 
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Health monitoring devices that were offered to participants included a Withings 
Nokia Body+® smart weigh scale, a Fitbit Charge 3® activity tracker and a Karie® 
automated medication dispenser developed by Ace Age. Apps for each of these devices 
were added to the participants’ smartphones so that they could also track and monitor 
their own data. Although activity logs for device usage were not recorded, health care 
providers could observe the measurements provided by the health devices through the 
LIDB thereby providing an idea as to frequency of device usage. 


Procedure 

The research team met with prospective participants and after obtaining informed con- 
sent, participants selected the equipment they felt they needed and completed the baseline 
interview. Interviews were conducted one-to-one with a member of the research team 
either in-person or virtually. Equipment selections were then verified and approved by 
the participant’s health care provider before being installed by the research team’s pro- 
grammer. Training on the equipment was provided by the research team programmer 
as well as the research coordinator. Technological literacy was measured at baseline 
through the demographics measure of the interview. Health care providers also received 
training on how to use the devices, CHR and LIDB by the research coordinator. The 
research coordinator contacted the participants monthly to check in on any potential 
technical difficulties and offered assistance if issues arose. Calls and emails were made 
to health care providers to offer support and troubleshooting with the LIDB, CHR and 
any general concerns regarding the devices. 

Focus groups with participants and health care providers were conducted separately 
to avoid one group from influencing the other. These were conducted in a group format 
prior to the COVID-19 pandemic but then switched to virtual groups or a one-to-one 
discussion via telephone or teleconferencing software. 


Instruments 

The following assessments tools were utilized for the semi-structured interviews con- 
ducted with participants: Community Integration Questionnaire - Revised (CIQ-R) [12], 
Short-Form 36 (SF36) [13], the Health, Social and Justice Service Utilization (HSJSU) 
questionnaire [14], EuroQol-5 Dimension-3 Level (EQ-5D3L) [15], and a Perception of 
Smart Technology Questionnaire. The latter was a researcher-developed measure that 
assessed participants' attitudes and opinions of the devices provided. Demographic and 
housing history data were also collected during each interview. The CIQ-R was the 
primary outcome measure. Open-ended qualitative questions regarding the use of the 
devices and software platforms were utilized during the focus groups and one-to-one 
discussions. 


Data Analysis 

The participants' health, housing, service utilization and community integration were 
assessed using the instruments. Descriptive statistics are provided for the outcome mea- 
sures. Due to the small sample size, comparative analyses across timepoints were not 
possible, and generalization beyond the sample is limited by its size. Qualitative data 
was analyzed using a thematic ethnographic approach [16] to ascertain collective expe- 
riences and implications. Data collected from the health devices and the Qnaires were 
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not obtained for analysis by the research team in order to maintain participant privacy 
and not to interfere with the patient-provider therapeutic relationship. 


3 Results 


A total of 13 participants were interviewed at baseline. Table 1 summarizes the demo- 
graphic characteristics of the sample. Nine participants completed the full 12-month 
study; two passed away due to natural causes before Time 2. At Time 3, one declined to 
complete the interview while the other participant could not be reached. No participants 
experienced housing instability throughout the course of the study. Two participants 
were eventually moved from group homes to independent apartments. The vast majority 
of participants were single or never married (n — 10), indicating that these participants 
may need additional support if they are living alone. 


Table 1. Demographics (N — 13) 


Age (mean (SD)) 43 (15) 
Sex 

Female 8 
Male 5 
Marital Status 

Single/Never Married 10 
Separated/Divorced 2 
Widowed 1 
Housing Status 

Independent Apartment 9 
Group Home 4 
Psychiatric Diagnoses 

Anxiety Disorder 9 
Mood Disorder 7 
Psychotic Disorder 5 
Disorder of childhood/adolescence 3 
Personality Disorder 3 
Substance-related disorder 1 
Physical Diagnoses 

Fibromyalgia 3 
Back Pain 2 
Diabetes/Endometriosis/Hemorrhoids/Hepatitis B/Hypertension/ Irritable Bowel 1 for each 


Syndrome/Peripheral Vasculitis/Polycystic Ovary Syndrome/Sleep Apnea/Ulcer (Foot) 
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In terms of technological literacy, the participants were asked at baseline to score 
their experience with devices (see Fig. 1). The sample overall indicated they were com- 
fortable with technology. Participants largely rated that they were comfortable with 
technology in general. Three participants reported they were “extremely comfortable”, 
six reported they were “comfortable”, and two participants each reported they were 
“slightly comfortable” and “slightly uncomfortable” respectively. 


Quantitative Findings 

Community Integration Questionnaire - Revised. The mean scores on all CIQ-R 
subscales and CIQ-R Total score across all 3 timepoints are reported in Table 2. The CIQ- 
R asked participants how often they completed certain activities and whether someone 
else assisted or completed these activities. 


Table 2. Community Integration Mean Scores 


Time 1 (N = 13) Time 2 (N = 11) Time 3 (N = 9) 
Mean (SD) Mean (SD) Mean (SD) 
CIQ-R Total (/35) 20.7 (5.69) 21.7 (4.67) 21.6 (4.99) 
Missing (n) 2 2 1 
Home Integration (/12) 8.81 (2.25) 9.56 (1.61) 8.63 (2.25) 
Missing (n) 1 2 1 
Social Integration (/10) 5.92 (2.06) 6.00 (2.21) 5.78 (2.86) 
Missing (n) 0 1 0 
Productivity (/7) 2.33 (1.78) 2.27 (2.10) 2.22 (1.79) 
Missing (n) 1 0 0 
Electronic Social 3.23 (1.48) 3.00 (1.95) 3.78 (1.79) 
Networking (/6) 
Missing (n) 0 0 0 


Short Form 36. The mean scores on all SF-36 subscales across all 3 timepoints are 
reported in Table 3. Subscales are scored on a scale of 0-100, with higher scores 
representing better health outcomes. 


Health, Social and Justice Service Utilization. A variety of health and social services 
were accessed throughout the study. Table 4 maps out the utilization of these services. 


EQ-5D-3L. The Visual Analogue Scale revealed participant's mean self-report ratings 
of their overall health, physical health and mental health from Time 1 to 3 on a scale of 
0 (lowest health) to 100 (best health) (see Table 5). 
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Table 3. Short form 36 mean scores 


SF-36 Subscale Time 1 (N= 13) |Time2(N=11) | Time3 (N=9) 
Physical Functioning 53.1 (34.3) 51.7 (33.3) 53.9 (37.6) 
Role limitations due to physical 51.9 (42.6) 47.5 (50.6) 36.1 (41.7) 
health 

Role limitations due to emotional | 46.2 (44.2) 23.3 (35.3) 55.6 (47.1) 
problems 

Energy/fatigue 43.5 (17.0) 34.3 (16.8) 49.4 (26.4) 
Emotional well-being 59.1 (19.2) 54.0 (21.9) 57.8 (21.6) 
Social functioning 52.9 (22.9) 46.3 (22.9) 59.7 (35.2) 
General health 41.9 (18.8) 38.5 (21.2) 44.4 (27.9) 


Perception of Smart Technology. Three key questions from the questionnaire are 
reported below. The first inquired as to whether participants felt the technologies 
improved their health care (see Fig. 1). This was asked at Times 2 and 3 after the tech- 
nologies had been implemented. The second focused on acceptability of the technologies 
in the home. The majority of participants responded favourably at Time 2 — Mixed (1), 
Mostly Satisfied (2), Pleased (4) and Delighted (4). At Time 3 there was a slight increase 
in positive ratings — Mostly Satisfied (1), Pleased (3), Delighted (5). 


The third key question focused on recommendations for what devices should be 
added to future interventions. The suggestions were evenly spread with a blood pressure 
cuff receiving the most recommendations (2), followed by a tablet, a smart television, 
smart glucometer, a Google Home® device and a newer model of Fitbit® (all 1). How- 
ever, the most frequent answer was "Nothing else/None" (3) as participants felt the 
devices offered as part of this study provided suitable coverage for their needs. 


Qualitative Findings 

Several themes pertaining to healthy living were discussed by the participants. Partici- 
pants noted that they were motivated to be healthier through exercising and maintaining 
a healthy weight. The health data on the apps on their smartphones allowed them to track 
their progress which provided a level of accountability. 

The scale especially um it, it, it allowed me to keep track of what was happening as 
far as weight and things like that went and um yea it didn't, it has motivated me um to 
start watching my diet and things like that. 

Health care providers also reported that their clients were becoming more motivated 
to use the devices provided to lead healthier lifestyles and, in some cases, assist with 
symptoms of pre-existing physical conditions. 

Uh, I definitely noticed, um, that she was able to maintain a healthy lifestyle, um, in 
term, or I guess a healthy weight. There was, you know, only small fluctuations in their 
weight, but it was nice to see that. Cause I know they, they talk about that a lot. Um, just 
wanting to, because they have another, um, illness related to their weight, like polycystic 
ovarian syndrome. So they were, it was more of like an important thing to focus on 
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Table 4. Services accessed at all timepoints 
Services in past month Response | Time 1 Time2 = Time3 
(N= 13) |(N211) (N=9) 

Seen a healthcare or social service provider at Yes 12 8 2 
their office No 1 3 7 

Sum 49 23 18 
Talked on the phone with health or social service | Yes 6 6 5 
provider No J: 5 4 

Sum 24 33 14 
Visited by healthcare or social service provider | Yes 6 3 2 

No 7 8 7 

Sum 69 19 41 
Past 6 Months 
Outpatient services at hospital Yes 6 4 2 

No 7 6 7 

Declined |0 1 0 

Sum 18 28 10 
Called Crisis Line Yes 3 5 4 

No 10 5 5 

Declined |0 1 0 

Sum 2 192 22 
Visited by Crisis Team Yes 1 1 1 

No 12 9 8 

Declined | 0 1 0 

Sum - 180 10 
Emergency Room Visits Yes I 8 4 

No 6 3 5 

Sum 22 40 12 
Been in an Ambulance Yes 6 7 3 

No 7 4 6 

Sum 10 15 6 

Table 5. EQ-5D-3L visual analogue scale scores 
Time 1 (N = 13) |Time 2 (N = 11) | Time3 (N 2 9) 
M(SD) M(SD) M(SD) 
Visual Analogue | Overall Health | 68.5(12.7) 64.1(15.8) 58.2(26.6) 
pale Mental Health | 68.1(15.2) 64.6(22.0) 67.8(17.3) 
Physical Health | 60.4(16.6) 54.6(18.5) 55.2(29.3) 
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Has the use of the home-based smart technologies improved your 
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Fig. 1. Perception of improvements to health care following the use of smart technologies. 


managing or getting, you know, getting to be a healthy weight so that the symptoms of 
that are minimized. 

Other participants noted benefits to their mental health through this new ability to 
be healthier and through the cognitive supports the technology offered. 

Thave issues with memory and being able to have reminders to drink water. Um, how, 
cause instead of feeling more lethargic and fatigued from not drinking water, um, I felt 
less lethargic, less fatigued. Therefore it increased my mood because I wasn’t sleeping 
all the time. And for me, a trigger for depression is sleeping all the time. 

It was noted that the technologies were also able to support mental health through a 
biofeedback approach with the devices monitoring physical activity. 

..and like as I say the, the Fitbit, especially the pulse um the heart rate um is, is 
really helpful because um if I'm having problems with my mental state sometimes, I need 
to look at, especially with anxiety, I need to look at my pulse and, and sort of be aware 
of it and help, it can help me bring it down... 

Enhanced communication was also seen as a major benefit of the study. Participants 
noted they were able to maintain communication with their friends, family, and health 
care providers during the pandemic. 

Um, well being connected with your family definitely helps you with your mental 
health and like being able to follow-up with appointments and stuff like that. Um, yeah 
and I also have been able to like, find resources about my mental health like on CAMH 
and stuff like that and um meditation resources as well. 

Many health care providers also spoke of the social benefits of providing participants 
with a smartphone but also noted that the technology enhanced communication with the 
care providers as well as flexibility in communicating with them. 

...Like some clients especially like with mental health, they find it easier for them to, 
instead of talking to someone over the phone, to text and communicate that way. 

There were numerous instances of participants noting that the reminders and prompts 
from the technology provided them with the support needed to maintain healthy lifestyles 
and live safely in their own homes. In particular, participants with the medication 
dispenser noted that they no longer missed doses. 
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Having the ability to monitor um the, the medication was a problem and the tech- 
nology just sort of took that problem away because it replaced the need for me to keep 
track of things. 

With technological interventions, there are also some limitations and barriers to 
overcome. One such issue is device reliability. There were some issues regarding the 
accuracy of their readings and difficulties with connectivity. 

Yeah, um, but when I switched phones it was very difficult to get um, to get my Fitbit 
to sync over Bluetooth. 

A precipitating factor for technical difficulties could be the lack of technological 
literacy and understanding how specific devices worked. Although monthly refreshers 
were provided by the research team, health care providers highlighted that frequent 
retraining would be recommended. 

I think there was, um, a few items that like the clients weren't really familiar with or 
like, knew how to use our troubleshoot, even though like, I don't know, maybe I'm just 
the younger generation. I know how to use it. Like the Fitbit. They don’t really know how 
to like sync it or look at their data on their, on the app, on the phones and stuff. 

It is also recommended that alternative devices are available for participants who 
require additional assistance with using technology. 

Like I said, I have hand tremors, so takes me awhile to get it to work. 

In terms of future improvements for commercialization, the health care providers 
suggested simplifying the approach to a single integrated database or as an app. 

You'll have to be streamlined through a maybe single-handed maybe app? ... that 
will probably be uh better, for future reference, and it could be uh quite heavily used 
then. 


Ethical Analyses. This project’s findings suggest that the use of the technologies 
advances equity and fairness by increasing access to care for people experiencing mental 
illness and physical comorbidities. During the COVID-19 pandemic, participants were 
able to access care from the safety of their own home using the technologies provided 
by the study. Another ethical advantage of this project was that it enhanced autonomy 
of participants, reducing their dependence on some health services such as by using 
the medication dispenser device. Overall, this project’s findings are promising from an 
ethical perspective and suggest the need for larger scale research on such technological 
interventions. 


Policy Analyses. It was felt that it would be purposeful to propose an amendment to 
the Assistive Devices Act to incorporate smart technologies as assistive devices for 
mental health. Devices aimed at supporting mental health are not currently covered by 
the Assistive Devices Act. Individuals with mental illness or agencies supporting this 
population would have to purchase the devices themselves. For individuals in other 
jurisdictions, amending current policy to provide funding for technological innovations 
in the home to support mental illness would be advantageous. 


4 Discussion 


There is a lack of literature on community-integrated research that utilizes a systems- 
level novel approach to health and smart technology. Smart technology interventions, 
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such as this study, should be designed to provide support a range of physical and mental 
health conditions [17]. This study positions itself as a foundation for future research 
among individuals with mental illness to build upon. As there was a small sample size, 
it would be difficult to argue that this intervention represents a definitive approach to 
mental health care, but provides an impetus for new research to cover some of the gaps 
currently in the literature. Data from the Perception of Smart Technology questionnaire 
supported that the technologies were well-accepted and used by participants of different 
age groups. As the majority of participants were single or never married, this intervention 
may be a needed support for individuals living along without immediate assistance. The 
connectivity of the intervention with care providers could help to mitigate potential risks 
that may arise due to isolation or a lack of communication. 

A key strength for this intervention was the use of non-clinical devices that are 
accessible to the public. Data from the clouds of these devices were able to be transmitted 
and collated in one database, the LIDB. Health care providers were able to access all 
the data in one location as opposed to individual databases or independent datasets. 
This allowed for a tailored approach where each participant could have a customized 
intervention. Participants were also able to connect their smartphone (either their own, 
or one provided as part of the study) to their devices and access the data themselves using 
the devices’ apps. An implication for future research is to measure the effects of smart 
technology among a larger sample with other mental and physical health conditions. 
It would be highly beneficial for potential researchers to track health data (i.e. weight, 
steps, activity, heart rate, etc.) empirically to assess the impact on health behaviours and 
physical condition. 

There is also the opportunity to utilize this intervention to support physical conditions 
that may benefit from activity and weight tracking and notifications from a health care 
team. This study found that smart technology was able to promote healthy lifestyle 
choices. Based on the qualitative findings, the devices acted as an accountability tool 
which provided encouragement and motivation for healthier living. Frequent observation 
and self-monitoring of health data using personal digital assistants and daily feedback 
messages have been linked with weight loss [18]. 

A systematic review by Liu et al. [19] reported there is no evidence that technology 
tracking and monitoring biometric data resulted in improvements to quality of life or 
disability. However, the qualitative findings of this study suggest that the feedback from 
the health adjunct devices and their respective apps on the smartphones can be helpful. 
The use of an activity tracker to encourage physical activity and exercise may have 
been the most contributory piece of equipment in this regard. Participants reported how 
the technologies supported mental health such as mood and anxiety through greater 
communication and the ability to monitor physical health more easily. Cognitive support 
was also described in assisting participants to remember tasks such as taking and tracking 
medications. There was some discussion among participants that the weigh scale and the 
activity tracker may have occasionally given inaccurate readings. Future studies would 
be advised to frequently check devices for accuracy during the course of the study and 
allay participant concerns. 

The COVID-19 pandemic meant lockdowns were enforced during the course of the 
study. Individuals with mental illness and physical conditions are especially vulnerable to 
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the negative outcomes of isolation and distancing [20]. The COVID-19 pandemic meant 
that virtual and telephone interviews were conducted instead. Although participants were 
satisfied with these arrangements, some participants were difficult to contact and required 
multiple sessions to complete the interview. The technician for this study provided 
technical support where possible but opportunities to visit the participant and check 
the devices were limited. 

Due to the small sample in this study, regression analyses were not conducted; thus, 
findings should be interpreted with consideration to this. However, the data collected 
from questionnaires are pertinent as it provides better understanding of the characteristics 
of individuals with mental and physical health diagnoses who may benefit from smart 
technology use. Notably, it was observed through questionnaire scores that there was 
high heterogeneity in our sample and in participants responses (e.g., frequency of service 
utilization). Further research into the use of smart technology with individuals who have 
mental illness with a larger sample is warranted. All participants from this study were 
recruited from the same, moderate-sized, city. A larger study with a variety of locations 
including more rural-based individuals may reveal different experiences and learnings. 


5 Conclusion 


This study establishes that a smart home technological intervention is a feasible, reliable, 
and safe way to provide additional support in the home. The provision of commercially 
available devices may provide a suitable alternative or an additional option for in-home 
support. These technologies can supplement healthy living behaviours which could lead 
to other health benefits such as weight loss, as supported by the qualitative analyses. 
Participants noted a self-reported increase in physical activity, diet tracking and greater 
access to mental health support. The COVID-19 pandemic likely impacted health, social 
and justice service utilization therefore studies exploring this, including this study, needs 
to be interpreted with caution. This study can act as a foundation for future research to 
build upon by exploring other applications and populations with this intervention. Future 
studies and technological innovations would be advised to offer in-home technologies 
that can be easily implemented into the living environment and to address gaps currently 
in the literature. 
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Abstract. Mobility is essential for older adults to keep a good level of socializa- 
tion, health and well-being. Still, aging is often accompanied by multiple mobility- 
related challenges. Hence, these mobility limitations make travel and use of public 
transport a big challenge. Numerous mobility trip planner tools can be found now 
days. However, they are not adapted to older adults’ needs and preferences. We 
discuss in this paper our attempt to develop a one-stop platform to help older adults 
in their mobility where, when, and how they want. We introduce our co-creation 
approach, our software architecture, as well as our first prototype. 


Keywords: Mobility - Older adults - Trip planner - Web application - 
Age-friendly trip planning 


1 Introduction 


The proportion of older adults has continuously growth over the years, as stated by 
World Health Organization [1]. A variety of difficulties and barriers related to mobility 
frequently appear with aging including: 1) Walking aspect: walking limitations, lower 
walking speed, loss of balance and incapacity to pass by a high slope road [2]; 2) Fears 
aspects: fears often arise with advancing age [3], including the fear of not getting off at 
the right bus stop, not getting on the right bus, making the use of public transportation a 
real challenge [4]; 3) Cognitive and Sensory abilities: vision and hearing deteriorate with 
aging [3] which may interfere with safe driving; 4) Financial aspects: advancing age is 
also accompanied by life transitions, including retirement, which leads to diminishing 
financial resources [3] that do not favor the use of taxis by older adults. Consequently, 
these difficulties and barriers affect older adults’ ability to move around safely and 
independently, prevent them from maintaining a social life as active as they wish. While 
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aging, promoting the autonomy represents an important vector for social integration 
of older adults and their participation in the activities of daily life. In this context, 
various planning transit tools around the world were developed to help people to get 
easily to their destination and provide information about various modes of transport. 
These transit tools are helpful for general public and basic use. However, they are not 
tailored to all older adults’ needs since they consider only a few aspects important to this 
population. We argue that a trip-planning tool providing personalized maps in terms of 
textual and graphical presentation and paths adapted to older adults’ physical and sensory 
impairments and cognitive capacities is much needed [3]. In this context, we introduce 
Mobilainés, Mobility as a Service (MaaS) platform which incorporates different modes 
and forms of transportation services, to help older adults move around where, when, and 
how they wish. 

The rest of the paper is structured as follows. Section 2 presents Related work. 
Mobilainés approach and mobility key needs are presented in Sects. 3 and 4. Section 5 
introduces Mobilainés architecture. Preliminary results are presented in Sect. 6. Lastly, 
the paper is concluded in Sect. 7. 


2 Related Work 


Trip planning tools were developed to help people reach their destinations through 
different modes of transport (e.g., car, taxi, bus, car sharing, bicycle, walking), give them 
details about the trip (e.g., directions, which bus to take, which station) and provide users 
the option of planning their trips whenever they want. Based on our analysis, most of 
the existing tools address the shortest/fastest path functionalities considering only one 
criterion (1.e., distance, time, or number of transfers), which make them not tailored to 
older adults’ needs and preferences, except for a set of tools that provide some options 
and functionalities that might be useful for older adults. A few number of tools address 
the walkability either by minimizing walking (transit!, moovit^), or by considering the 
maximal tolerated walking distance and speed of the walk (Mobilitéit?, alternéo^). The 
prototype mPASS [5] customizes maps and paths to each user's needs and preferences, 
based on the user's profile and location. mPASS [5] provides users with accessible paths 
taking into consideration accessibility barriers and facilities like stairs and zebra crossing. 
Path2.0 [6] suggests accessible paths to users with disabilities by providing users with 
the shortest/fastest path as well as the path that was frequently utilized by other users with 
the same disabilities. Transit (See footnote 1) is a mobile application proposing paths 
without stairs and avoiding slopes; while ViaNavigo?, RATP® and Google maps’ suggest 


! Transit trip planning tool:” https://transitapp.com/ (accessed May 16, 2022). 

? Moovit Urban Mobility App in Sherbrooke, QC,” https://moovitapp.com/sherbrooke qc-3064/ 
poi/en (accessed May 16, 2022). 

? Mobilitéit trip planning tool: https://www.mobiliteit.lu/fr/ (accessed Oct. 19, 2021). 

^ Alternéo trip planning tool: https://www.alterneo.re/ (accessed Oct. 19, 2021). 

: Vianavigo trip planning tool: https://me-deplacer.iledefrance-mobilites.fr/itineraire (accessed 
Oct. 19, 2021). 

6 “RATPtrip planning tool for Paris-France: https://www.ratp.fr/ (accessed Feb. 08, 2021). 

7 Google Maps App: https://www.google.ca/maps (accessed Oct. 19, 2021). 
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accessible paths for people with reduced mobility using a wheelchair. Furthermore, few 
tools give information about the crowd, in other words, road traffic (Google maps (See 
footnote 7)) and bus crowd (Google maps (See footnote 7) and RATP (See footnote 
6)) that remains to be a useful information especially for older adults who want to 
avoid crowded places and roads. In the aim of encouraging active and ecofriendly paths, 
Citymapper? calculates respectively the burned calories and the amount of CO» emission. 
Additionally, information about weather was given by Mobilitéit (See footnote 3), OC 
transpo?. To conclude, the presented trip planning tools don't consider the totality of 
older adults' needs and preferences. Therefore, there is a need for an approach that takes 
the diversity of older adult needs in consideration for building the trip planner, which 
was the first motivation behind our Mobilainés project. 


3 Mobilainés Approach 


Mobilainés is Mobility as a Service (MaaS) e-tool, in other words, a one-stop platform 
transport service combining different modes of transport and various forms of transport 
services to help older adults move where, when, and how they want. In order to meet 
older adults' needs, Mobilainés is based on a living lab co-design approach, which 
facilitates the collaboration between stakeholders from various sectors to create, validate 
andtest new technologies, services, products and systems in real-life contexts, and makes 
the interaction with older adults easier [7—9]. For this purpose, a steering committee 
composed of 8 stakeholders from the public, scientific, and private sectors as well as 
older citizens (n — 8) was formed. Like any other living-lab approach, Mobilainés project 
comprises three phases, each with several iterative subphases (More details on [10]). 


1) Exploration: Define older adults’ mobility key needs and preferences: The aim of 
this first phase of the project is to identify, document, and understand older adults’ 
needs and preferences in terms of mobility by conducting literature reviews and 
codesign workshops, including 1) identification of older adults’ mobility barriers and 
facilitators especially in our city of deployment (i.e., Sherbrooke, Quebec, Canada), 
2) Highlight the gaps and understand how the existent transportation services and 
trip planning tools don’t meet older adults’ needs and preferences; 3) Define criteria 
to consider and functionalities to propose for Mobilainés to be adapted for older 
adults. Results of the exploration unable to understand the required profiles of the 
older adult in building the trip planner. 

2) Experimentation: Co-creating the prototype of the one-stop platform. The aim of 
this phase is to propose an adapted and personalized paths to older adults. Concretely, 
this phase focus on three elements: 1) Data acquisition: Data related to identified 
needs and preferences and considered criteria is gathered from various sources. We 
process this data and use it to meet the identified needs and preferences. 2) Trip plan- 
ning: Later criteria priority weight is defined, which attribute a level of importance 
to each considered criteria. Then various multicriteria planning algorithms are tested 
and evaluated. The selected algorithm will be used as the routing engine that takes as 


5 Citymapper trip planning App. https://citymapper.com/ (accessed Oct. 19, 2021). 
? OC Transpo trip planning tool: https://www.octranspo.com/en/ (accessed Feb. 08, 2021). 
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input the departure, destination address, trip time and the considered criteria to give 
as output the adapted path. 3) User interface: The user interface implementation 
is done using an iterative process. Therefore, usage scenarios and mock-ups (static 
designs of the web application) were produced to present potential functionalities 
(content and interface). 

Evaluation: Older adults with various physical and cognitive conditions and multiple 
mobility preferences will participate in the usability tests of Mobilainés platform. 


Key Needs to be Addressed 


The Mobilainés team run a survey about older adults’ mobility needs and preferences, to 
better address the requirement of the older adults in the development of the trip planner. 
Following is a summary of the retrieved six key values that highly impact older adults’ 
mobility and needed to be included in the trip planner (more details can be found on 
[10]), with examples of how we can address them in Mobilainés: 


1) 


2) 


3) 


4) 


Health: Older adults are generally looking for a path that takes into consideration 
their physical, mental, spiritual state and social well-being. Indeed, daily mobility 
is a kind of exercise for older adults that helps them undertake an active and healthy 
lifestyle [11]. 


e Example on how we address Health is by providing adapted walking options, 
e.g., paths that avoid slopes, recommend paths passing by benches to have a rest, 
consider walking speed in path planning. 


Safety: for the protection of the physical, emotional, and psychological integrity of 
the older adults. In fact, personal security concerns can install a feeling of fear from 
crime [12], accidents, harassment, but also misbehavior of staff [13]. This anxiety 
prevents older adults from using public transport. 


e Example on how we address Safety is by helping users plan their return before 
sunset (so they won’t be out in darkness), and by consider less crowded roads. 


Quality of life (QoL): Mobility is significantly associated with quality of life (e.g., 
psychological benefits) among older people [14] and a lack of mobility options or 
transportation has an important impact for older adults’ satisfaction and sense of 
personal well-being [11, 14]. 


e Example on how we address QoL is by including agreeability for leisure trips, 
e.g., by recommending paths passing by parks. 


Equality: Promote mobility/public transport for all older adults with the same con- 
sideration is primordial. Unfortunately, the mobility/public transport services are 
not accessible for everyone, and it seems that some regions or sub regions are less 
served [15]. In certain cases, this is linked to language and cultural barriers [13]. 
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e Example on how we address Equality is a) by providing diverse mode of trans- 
ports in respect of various levels of disability (not only public transport), b) by 
suggesting affordable alternatives for people with limited budgets, as well as c) 
by addressing language and cultural barriers, e.g., the user interface is accessible 
via two languages French and English (The most spoken languages in Quebec - 
Canada) 


5) Autonomy: The ability to take independently transport (and move around) plays a 
crucial role in older adults’ freedom and independence, and “the access to public 
transport can help older adults to avail themselves of goods, services, employment 
and other activities” [16]. 

e Example on how we address Autonomy is by simplifying the user interface, and 
by providing various alternatives of paths so that the user can choose freely the 
path he wants to follow. 

6) Eco responsibility: It is the mobility choices that aim limit impact on the environ- 
ment. In fact, environmental preferences impact older adults’ choice of a mode of 
transport [17]. 

e Example on how we address Eco responsibility is by providing older adults 
with information about CO2 emissions of each proposed path, and by promoting 
eco-friendly options, e.g., bike, public transportation. 

5 Mobilainés Architecture 

OSM data 2 
U 
builder Engine 

Additional data: 

> Google maps 
> Open data provided 
by Sherbrooke city 
>Openweather data 


Fig. 1. Mobilainés architecture 


Creating an adapted trip planner tailored to older adult needs and preferences (that 


considers the six described key needs) requires 3 key components /Layers: 1) Data 
layer: where we basically get data from various data sources to consider the older 
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adults needs, 2) An age-friendly user interface: older adults are not familiar with 
new technologies, therefore it's important to provide an easy to use and adapted user 
interface, 3) Mobilainés engine: an engine that is based on two main components: 3.2) 
Profiling: a key element for a better personalization and adaptation of a trip planner is 
to better creating an adequate profiles, and 3.2) Planification engine: this component is 
responsible of a multicriteria algorithm optimization that considers various and multiple 
criteria and provide the best compromise of all (Fig. 1). 


5.1 User Interface Layer 


As people age, they inevitably experience certain physiological and cognitive changes 
that make the use of technology challenging [18]. Nevertheless, there are also older 
adults who aren't as comfortable or familiar with technology [19] therefore Mobilainés 
platform should be designed in a way that makes of it accessible and adaptable to older 
adults. Before designing Mobilainés, we first analyzed the existing Trip planning tools 
and tested trip planning mobile applications by 6 older adults aged over 65 years via a 
workshop. Results of the workshop confirmed that the existent tools have strengths and 
weaknesses regarding user interface adaptability to older adults. Most of the evaluated 
trip planning tools lack users’ support and do not have indications where the user is in 
the process of planning his trip, most of them require multitasking what makes them 
not adapted. To better accommodate the six key needs (Sect. 4) we opted for building 
user interface by respecting the following four design principles: 1) Easy to use: The 
website's text is adjustable, and it provides large buttons [20, 21]. Our user interface 
provides an easy navigation by having a flowchart that shows the step the user is in 
within his path planification. Furthermore, each page contains navigation buttons to go 
to the next step and go back to the previous step [22]. 2) Consistency: The designed 
user interface uses icons used by other trip planning tools for the same purposes [22] 
(e.g., favorite icon, icons for the walking, bus). 3) Help and support: To support older 
adults with digital literacy in using our platform, we also provide a contact phone call 
to reach an agent who is there to help in planning the trip [23]. 4) Minimizing memory 
load the designed interface provides a summary page of validation to make sure that the 
older adult is aware of her/his inputs. It also shows recent trips for quick trip planning 
and give her/him the option to save a trip that can be reused. 


5.2 Mobilainés Engine Layer 


Mobilainés engine is a server application that exposes numerous methods for routing, 
signing up users, updating profiles, etc. The Mobilainés engine is based on two main 
components: 


5.2.1 Profiling 


Mobility needs and preferences are diverse from an older adult to another they depend on 
the individual's health, psychological and financial conditions. Therefore, it's necessary 
to create a profile that captures this diversity and includes the user characteristics, phys- 
ical capacities and limitations, mobility behavior, etc. By adding a profile component to 
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Mobilainés trip planner, we can include more information on user needs and preferences 
in the planning of the trip, and hence propose personalized and adapted paths that are 
as close as possible to what older adults is looking for. Adding information on Mobi- 
lainés trip planner use will also allow to learn from user mobility behavior to improve 
Mobilainés and propose better itineraries in a potential second version. 


5.2.2 Planification Engine 


The planification engine is a major component of our proposed Mobilainés engine. 
Indeed, there are multiple open-source software tools and APIs that were developed in 
the purpose of trip planning and that includes routing engine. The two most prominent 
open-source solutions are NAVITIA ? and OpenTripPlanner!! (OTP). We opted for OTP 
due to its powerful trip planners and the parameters taken into consideration in path 
planning (e.g., stairs reluctance, preferred routes, and banned routes). OTP includes a 
graph builder and a basic routing engine based on A-star and Range Raptor algorithms. 
Our additional layer focuses mainly on considering multiple criteria and consequently 
build a multi-criteria route plan algorithms. 


5.2.3 Data Layer 


The data layer is the architectural unit that provides our routing engine with all the 
necessary information and data to return the optimal route to the user. Data layer is 
linked to the platform users' profile, preferences, route history, and any other piece of 
information that proves useful to our routing algorithm. Various data sources are needed 
for a trip planer adapted to older adults’ needs, including: 1) Geolocalization data to 
collect data about streets and amenities; 2) City data (of Sherbrooke in our case)". 
More cities now have various open-source databases with rich information that can be 
beneficial for older adults' trip planning, e.g., information on snow removal that can help 
plan a safe trip; some that enables avoiding icy sidewalk in winter; and some that makes 
possible considering road to avoid road segments with affected by the work; 3) Transit 
data: Public transit agencies provide their transit data including trip updates, service 
alerts and vehicle positions and bus traffic in the general Transit Feed Specification — 
Realtime (GTFS-RT) specification [46]; and 4) Weather data (OpenWeather in our case) 
are used to consider sunset time, temperature and snowstorms predictions to propose 
safe trips. 


6 Preliminary Results - Moblaines First Prototype 


To demonstrate that the architecture discussed having a practical potential, we developed 
the first prototype for Mobilainés project, with the following components: 


10 Navitia trip planner platform: https://navitia.io/ (accessed Apr. 19, 2022). 

B "OpenTripPlanner: http://docs.opentripplanner.org/en/latest/ (accessed Jun. 02, 2021). 

12 «Sherbrooke City Open Data." https://donneesouvertes-sherbrooke.opendata.arcgis.com/ 
(accessed Apr. 19, 2022). 
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6.1 Data Acquisition 


We use in our first prototype: 1) OpenStreetMap data for geographical data, to collect 
street data, data about public toilets, benches, etc., and 2) GTFS data for transit, including 
trip updates, service alerts and vehicle positions, supplied by the bus company of the 
city of Sherbrooke. 


6.2 Engine 


We use in our first prototype OTP engine, for non-transit trip (walking, bike, car) plan- 
ning. The results generate a list of itineraries using A-star search, an algorithm that 
extends the Dijkstra's algorithm for better computation complexity. It uses heuristic 
evaluation function (in this first prototype we use Tung-Chew heuristic [24]) to calcu- 
late the costs of each neighboring node. However, for transit paths, the path is segmented 
into three segments [25]: 1) Access from the origin to transit stops, 2) egress from transit 
stops to the destination where we use. A star search algorithm (same as non-transit trips), 
and 3) transit service connecting the two using the Multi-criteria Range Raptor algorithm 
[26]. For two given bus stations, it computes all Pareto-optimal journeys minimizing the 
arrival time and the number of transfers made—between them. RAPTOR [26] is round 
based, in other words, it operates in rounds and processes each route of the network at 
most once per round. 


6.3 User Interface 


We built our first prototype of the user interface for trip planning according to the result 
of our Exploration co-creation phase, putting in mind the usability of the interface by 
our older adults. The first prototype of the user interface is composed of four pages: 


Path definition 


Path definition 


Recent paths 


Fig. 2. Path definition page 
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1) Path definition (Fig. 2. Path definition page), where the user adds his departure 
and destination, he can do that either by typing an address or by selecting an area in 
the map. There is also an option to add addresses to favorites. Finally, the user sets the 
timeframe of the trip, to either leave now, arrive by, or depart at. This page also keeps 
track of the user’s recent planned paths so they can be accessed directly. 2) User options 
(Fig. 3. User options page.), on this page the user selects his needs and preferences (this 
page can be filled in automatically by the information stored in the user profile). 3) the 
third page (Fig. 4) shows all itineraries returned by the routing engine that meets the 
user’s selected needs and preferences so that he can select the path to follow. 4) The 
fourth page (Fig. 5) is for validation, we make sure that the user is aware of what he 
chose as a departure/destination address, timeframe and selected needs and preferences. 
Finally (Fig. 6) the user can visualize the chosen path on a map, with directions, and 
information about this path. 


User options 


Fig. 3. User options page. 


Available paths 


Recommended paths 


Walk, Bus 16 


Other paths 


Walk, Bus 8 Walk, Bus 9 Walk, Bus 14 
iQ iQ #Q 
m nac nma [A 


Drs rii Orbe Fb 
wok: Sore Vinc nm 
Foes 225 


Fig. 4. Available paths page 
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Directions 
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9 777mm 
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Path information db doudy $2: Crowded f) Save current trip 
Print with directions 


Fig. 6. Path’s details page 


7 Conclusion 


We introduced in this paper Mobilainés, a trip-planning tool adapted to older adults 
needs that consist of one-stop platform transport service combining different modes 
of transport and various forms of transport services to help older adults move where, 
when, and how they want. We also introduced our adopted a co-creation process in 
Mobilainés project. The process allowed to highlight the six key needs in mobility (i.e., 
Health, Safety, Quality of life (QoL), Equality, Autonomy, and Eco responsibility) and 
the technological services to address these needs. We also presented in the paper the 
Mobilainés software architecture to implement the desired services, as well as the first 
prototype. Usability tests are planned to evaluate the prototype and identify gaps to be 
filled in the next version (that includes a multi-criterion optimization algorithms). 
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Abstract. Patients discharged from hospitals following the onset of an acute 
illness or injury rendered with disabling conditions require systematic medical- 
based and rehabilitation-focused sports and exercise programs accessible in their 
communities. This proposal aims to build a data-driven smart health system that 
allows people with disabilities to continuously improve their health by alleviating 
modifiable factors, including architectural barriers and related challenges follow- 
ing discharge from an inpatient hospital or rehabilitation course. Our goal is to 
promote a multi-ministerial data-driven innovative medical exercise system using 
a digital living lab platform as a testbed program to provide lifestyle exercise 
and physical education for community-dwelling individuals with disabilities. The 
pilot program of services will be rendered at the living lab center of the National 
Rehabilitation Center, equipped with data servers for storing accumulated perti- 
nent information and continuous data acquisition. We envision an encrypted data 
collection and acquisition system, whereby newly acquired data will be merged 
with data information from original records of individuals generated during the 
inpatient hospital course. 


Keywords: Smart medical - Rehabilitation sports and exercise - Data - Living 
lab - Disability 


1 Introduction 


Upon completing an inpatient rehabilitation course, most patients discharged from the 
hospital face medical complications, which places them at high risk for readmission 
[1, 2]. Without proper exercise programs, equipment, and guidance by trained exercise 
professionals, to promote their medically proven health care, they are falling into the 
abyssal valley of death between hospital care and community sports. This is especially 
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true for individuals with a spinal cord injury or stroke, whereby the rendered paralysis 
usually results in varying degrees of impaired mobility [3, 4]. 

In the absence of an appropriately equipped environment conducive to promoting 
self-directed care in health and medical management, individuals with disabilities are 
denied equitable participation in many aspects of community life. And thus, after receiv- 
ing rehabilitation treatment from an inpatient hospital setting due to an illness or acci- 
dent that results in disabling conditions, patients require a systematic medical-based and 
rehab-focused sports and exercise program within their dwelling communities. 

These programs should be comparable to or mimic the services rendered from the 
perspective of an institutional setting. It should also serve as a vital component in the 
overall health management and continuity of care within a community environment. 
However, local communities lack the necessary infrastructure—workforce, space, equip- 
ment, and devices— encompassing a program of services that focuses explicitly on 
smart medical-based and rehab-focused exercise and sports activities catered to these 
individuals [5]. 

Furthermore, from an economic perspective, studies have substantiated cost-benefit 
aspects based on total annual medical and healthcare expenses for individuals with 
disabilities by comparative studies analyzing expenditure outcomes between individuals 
with disabilities who engage in some form of regular exercise activity as opposed to those 
who did not regularly participate in exercise programs [6, 7]. 

And thus, as the marginalization of the disabled population—inequity of essential 
healthcare amid a public health crisis—continues to be the social norm globally [8], such 
a provision for this population undoubtedly yields equitable opportunity in promoting 
their health and well-being as that of the non-disabled members of their communities at 
large [4]. 

We intend to build a system that allows people with disabilities to continuously 
improve their health by alleviating modifiable factors and related challenges after being 
discharged from the hospital. This study aims to promote a multi-ministerial data-driven 
smart health system using a digital health living lab platform as a testbed program 
for providing lifestyle exercise and sports-related physical education for community- 
dwelling individuals with disabilities. 


2 Paper Preparation 


The common aim was to promote the activities and exercises of people with disabilities 
in the community through joint efforts of multi-ministries to integrate resources and 
infrastructure from the medical, technological, economic, and social sectors. Through a 
governmental competition among 108 candidates, strategic planning with a large grant 
was awarded to the National Rehabilitation Center (NRC) in the Ministry of Health & 
Welfare by joint efforts from the three related ministries: Ministry of Culture-Sports & 
Tourism, Ministry of Science & Technology, Information & Communication, and For- 
est administration. To fulfill the tasks, appropriate requests for proposals (RFP) were 
designed by each ministry. 

The collaboration of tasks and assignments span broadly under the tutelage of four 
governmental ministries: 1) the Ministry of Health and Welfare, which oversees medical 
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exercise services and its distribution as a leading role; 2) the Ministry of Culture, Sports 
and Tourism, which oversees exercise programs in the community; 3) the Ministry of 
Science, Technology Information and Communication (ICT), which oversees exercise- 
related data service and information technology. 

The Ministry of Health & Welfare, NRC, has announced 21 projects as a leading 
role. The individual projects are categorized into four disciplines: 1) “Partnership and 
Collaboration,” 2) “Equipment and Biometric Technology,” 3) “Data Integration,” and 
4) “Exercise Programs and Services.” The assigned projects by recipients of various 
academic institutions and business corporations are summarized as a figure which is not 
included herein but is available as a supplement. Each task consists of a consortium of 
universities and business corporations collaborating on delegated sub-tasks, ultimately 
establishing the NRC ‘Smart Rehabilitation Exercise Living Lab Center’. 


2.1 Roadmap and Plan of Action 


The defined roadmap for this multi-faceted endeavor of building the smart rehabil- 
itation exercise living lab center requires integration with collaborative efforts from 
multi-ministerial forces. The collaborating entities will work toward data acquisition 
and service connection using input from the public and private organizational resources. 
Field application within the designed living lab center will also require expert guid- 
ance from clinicians, therapists, and allied health professionals from disability sports 
and physical education arenas. The action plan spans a three-year timeline focused on 
accomplishing three broad aspects. 

The first year (2021) was dedicated to the detailed design of the smart medical living 
lab platform. This was achieved by collecting opinions from the multi-ministerial coun- 
cil and advice from expert advisory groups. Participating entities successfully received 
institutional review board approval for clinical trials involving human participants. Plat- 
form integration was also accomplished by connecting with the Ministry of Health and 
Welfare representatives to oversee medical services distribution, the Ministry of Culture, 
Sports and Tourism to supervise the exercise and sports activities, and the Korea Forest 
Service. 

In 2022, Year 2, the focus was to set up and execute operation plans for the smart 
medical living lab. The center will be set up and housed at Korea National Rehabilita- 
tion Center. The prototype module development and integration of various established 
platforms will be connected under the multi-ministerial linkage efforts. The recruitment 
efforts and enrollment of study participants for the preclinical trials with testbed services 
of the program activities will get underway. 

The focus in the final period, Year 3, is to verify and stabilize operations efforts 
of the disability exercise center via expansion establishment of the model nationwide. 
The final evaluation and assessment will be performed along with data linkage with the 
multi-ministerial entities. 

The described pilot services in physical education and program of activities will be 
rendered at the living lab center of KNRC. The center will be equipped with data servers 
to store continuously acquired data and accumulate pertinent information. We envision 
an encrypted data collection and acquisition system, whereby newly acquired data will 
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be merged with data information from original records of individuals generated during 
the inpatient hospital course. 

Twenty-one tasks have been delegated across various institutions and corporations. 
These include four assignments for systems, seven for equipment and device devel- 
opment projects, five studies in physical education and exercise programs, and five 
assignments in data linkage. 


2.2 Pilot Test Run 


We acquired preliminary data using the above data set and conducted a testbed run. The 
study sample (n = 7) comprised five athletes and two coaching staff from the Korea 
Wheelchair Rugby Association. All participants were of male gender and had a varying 
cervical level of neurological injury (C5-C6-C7); one with thoracic level injury (T3). 
The para-athletes were initially examined by a staff physiatrist trained in spinal cord 
injury medicine. The basic clinical characteristics of the participants were collected 
followed by an assessment of “functioning status’ using the 6-min Walk Test (6-MWT) 
via wheelchair propulsion. 


3 Results 


The dataset we envision collecting via purchased equipment, manufactured devices, 
apparatus, and survey questionnaires from personal interviews comprises 208 pieces 
of information. This dataset is broadly categorized into three sub-datasets: 1) Personal 
information (PI), 2) Evaluation Data, and 3) Exercise Data. In the Evaluation Data cat- 
egory, it consists of four sub-categories. These include medical and community records 
about general health conditions, which were evaluated, 1) during hospitalization (dH), 
2) in clinics after hospitalization (1C), 3) from a personal trainer (PT), and 4) National 
Insurance in South Korea (NI). These periodical evaluations are able to determine each 
individual's health condition at the time of clinical evaluations, which would be com- 
pared to community exercise intervention programs. The Exercise Data category consists 
of two sub-categories. These include 1) the biometrics of exercise (BE) using physiolog- 
ical sensors, e.g., heart rate, oxygen saturation during the exercise, and 2) the amounts 
of exercise (AE), frequently described by parameters of Frequency (# per a week), 
Intensity (Exertion and speed), Time (mins per one time), Type (Aerobic/Anaerobic, 
Cardiac, Strengthening, Resistance, etc.). If there exist intersection parameters in evalu- 
ation/exercise sub-categories, each recorded time would be different to discriminate the 
health conditions. 

These evaluation data can be further categorized into six sub-datasets in terms of 
data characteristics. Table 1 summarizes the demographic information (#1—10); Table 
2 shows (1) Body composition (#11-30), (2) Administrative medical and health data 
(3331—40: medical and surgical histories; lab draws; diagnoses), (3) Functioning status 
(#4 1-90: physiology; biomechanics; body function test), (4) Quality of Life and phys- 
ical assessment (291—101: social; mental; general). Table 3 shows the exercise data 
(3102-130) summarizing the exercise performance in the program of activities. These 
data were examples of parameters collected from the manufactured devices: balance & 
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coordination exercise; anaerobic exercise; aerobic exercise, which is generated by the 
exercise equipment manufactured by the participating companies in current projects. 


Table 1. Clinical characteristics (Personal Information (PI)) 


Code Data Units Collection Method 
1 Demographics | HD0001 | Name Initials (KJB) | PI 
2 | HD0002 Age Yrs PI, dH 
3 | HD0003 | Sex/Gender M=0;F=1 PI, dH 
4 | HD0004 |Date of Birth |- PI 
ra HD0005 Marital Status |N=0;Y=1 |PI 
6 | HD0006  Offspring(s) #Count PI 
33] HD0007 Employment |N=0;Y=1 PI 
status 
g «| HD0008 EtOH & Tobacco |N=0;Y=1 PI 
use 
o | HD0009 Disability Level |- PI, dH 
category 
40 | HD0010 | Social History PI 


3.1 Results of Pilot Study 


The participants were all male and had a mean age of 29.7, a range of 19 to 46, and 
a median age of 25. One of the participants had a thoracic level injury (T3), while 
the remaining six were cervical level injuries. These and other basic characteristics are 
summarized in Table 4. The parameters measured in the 6- MWT included heart rate 
(HR), rate of perceived exercise (RPE), respiratory failure (RF), metabolic equivalence 
in exercise test (METs), maximal oxygen consumption (VO2/kg(ml)), minute ventilation 
(VB), respiratory exchange ratio (RER(R)), distance propelled in meters, and reason for 
discontinuing when applicable. The measurement also included resting BP and HR and 
post-exercise BP and HR. The acquired data are reported in Table 4 (Personal Information 
as "Clinical Characteristics"), however Table 3A (Functioning Status as measured by 
the *6-MWT") data are not reported herein but are available as a supplement. 


Table 2. Evaluation data 
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m HM0001 Height cm dH, PT 
12 HM0002 Weight kg on 
13 HM0003 Blood Pressure mmHg iC, dH 
14 HM0004 Body Mass kg/m? no 
%MVIC, 
15 8 HM0005 EMG EV iC 
16 E HM0006 Body Water L iC 
17 E HM0007 Protein kg iC 
i8| ô HM0008 Minerals kg iC 
i | = HM0009 Body Fat kg iC, dH 
20-| ® HM0010 BMI kg/m? iC 
21 HM0011 Body Fat (96) % iC 
22 HMO0012 | Abdominal Fat (%) % iC 
23 HM0013 Lean Body Mass iC, dH 
24 HM0014 Bone Density iC, dH 
25 HM0015 Muscle Mass kg iC, dH 
26 HMooig|  ^ndrod/Gynecoid iC, dH 
27 HM0017 AG Ratio iC, dH 
28 HM0018 Heart Rate bpm iC, dH 
29 HM0019 Respiratory Rate bpm iC, dH 
30 HM0020 O2 saturation SpO296 iC, dH 
ASIA 
31 HM0030 Diagnosis Impairment iC, dH 
Z Scale 
32 é | HM0031 Onset Date - iC, dH 
33 | $| & | HM0032 Cause of Onset - iC, dH 
341 A| A [HM0033 Type of Disability = iC, dH 
S Past Med/Surg : 
35 | 3 HM0034 iC, dH 
o) History 
36 F HM0035 Hospitalizations iC, dH 
Basic Metabolic iC, dH 
a _ dH, 
7g HM0036 Parel E 
o " 
T Complete Blood iC, dH, 
38 HM0037 Cont Ni 
39 HM0038 Lipid Panel ~ 
40 HM0039 HbAIc P 


(continued) 
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Table 2. (continued) 


41 HF0001 ROM degrees dH 
Grade . 
42 HF0002 MMT 123.4,5 iC, dH 
43 HF0003 Timed Up & Go sec iC, dH 
44 HF0004 ]Om walk test sec dH 
45 HF0005 FMA 0-100 iC,dH 
46 E HF0006 FAC 0-5 iC,dH 
47 | a a HF0007 Brunnstrom Stage 1-6 iC,dH 
a S8 
as |5 3 | HFo008 toga 0-56 iC, dH 
^ 8 Scale 
49 | £ pà HF0009 6-min walk test m 1C, dH 
50 5 3 HF0010 SPPB 0-12 iC,dH 
9 ea Multifunctional . 
51 E HF0011 Reach Test cm iC,dH 
52 HF0012 MBI 0-100 iC,dH 
53 HFo013 |  Sitto-stand 10 | — suo ig iC,dH 
times 
54 HF0014 Sit-to-stand 60 count/0- iC.dH 
sec 10 
55 HF0015 PCI iC, dH 
56 E 3 HB0001 Peak Torque ft-Ibs iC, dH 
57 fa} E HB0002 Peak TQ/BW % iC, dH 
58 2 HB0003 Total Work J (N.m) iC, dH 
Total Work % 1C, dH 
59 HB0004 Body Weight % 
60 HBO0005 Max Rep Tot N iC, dH 
Work 
61 HB0006 Coeff.of. Var iC, dH 
62 HB0007 AVG. POWER Watt(W) iC, dH 
63 HBOO008 Acceleration mis iC, dH 
Time 
64 HB0009 Deceleration time m/s iC, dH 
65 HB0010 Average Torque N.m iC, dH 
66 HBO0011 Bia. % iC, dH 
comparison 
67 HB0012 AGON MEAG % iC, dH 
Ratio 
68 HB0013 Wheelchair m/s iC, dH 
Speed 
Pushing Force ; 
69 HB0014 (Wheelchair erpo) N iC, dH 
Torque N.m or : 
" HEUS (Wheelchair ergo) J/rad iCpaH 
Torque (Arm N.m or , 
71 HB0016 Ergo) Jrad iC, dH 
72 HB0017 Force (forceplate) N iC, dH 
Moment š ; 
73 HB0018 (forceplate) kg-m iC, dH 
3D tracking ; 
74 HB0019 (VICON, IMU) iC, dH 
3D Angle 5 : 
75 HB0020 (VICON, IMU) iC, dH 


(continued) 
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Table 2. (continued) 


76 HPO0001 VO2max d oe iC, dH 
TI HP0002 FVC L iC, dH 
78 HP0003 FEVI L iC, dH 
Maximal iC, dH 

79 HP0004 Voluntary L 

5s Ventilation, MVV 
80 È HP0005 VCO2 l/min iC, dH 
81 8 HP0006 VO2 l/min iC, dH 
2 p 7 
82 £ HP0007 E l/min an 
83 HP0008 O2 Pulse ml.beats iC, dH 
84 HP0009 Tidal volume, TV L iC, dH 
85 HP0010 VE Max L/min iC, dH 
86 HP0011 Respiratory Rate, iC, dH 
RR 
87 HP0012 Breathing % iC, dH 
Reserve 
88 HP0013 VD ml iC, dH 
89 HP0014 End-Tidal O2 iC, dH 
90 HP0015 End-Tidal CO2 iC, dH 
91 E HG0001 PAR-Q iC, dH 
92 E HG0002 FRAX iC, dH 
93 Ó HG0003 MNA-SF iC, dH 
94 HG0004 K-PASE iC, dH 
95 = HG0005 MOCA-K iC, dH 
96 E HG0006 Depression Test 1-15 iC, dH 
97 S HG0007 Pain Scale iC, dH 
98 HG0008 FSS (Fatigue iC, dH 
Severity Scale) 

99 | $| 2 &| HG0009 EQ-5D-3L 5-15 iC, dH 
100| 8| = E HG0010 EQ-5D-VAS 0-100 iC, dH 
101 Q HG0011 SF-12 iC, dH 
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Data from 
Prototype 
Devices 
Developed 


Table 3. Exercise data 
Aerobic HR0001 | N of Number AE-F 
Exercise Performance count# 
HR0003 | Force Kg BE 
HR0004 | Angular rad/s BE 
Velocity of 
Arms 
HRO0005 | Angular mm/s? BE 
Acceleration of 
Arms 
HR0006 | Oxygen SpO2 BE 
Saturation 
HR0007 | Heart Rate bpm (beat per | BE 
minute) 
HRO0008 | Performance t/m/s AE-T 
Time 
HR0009 | Speed/Distance | km/h AE-I 
(Ergonometer) 
HR0010 | Pedal RPM rpm AE-I 
HROO11 | Calories kcal AE 
HR0012 | Total Time t/m/s AE-T 
Anaerobic HR0013 | Joint Position (x,y,z) BE 
Exercise HR0014 | Joint Angle o BE 
HR0015 | Isometric Force | N AE-I, BE 
HR0016 | Isokinetic Power |W AE-I, BE 
HR0017 | Isotonic Power |W AE-I, BE 
HR0020 | Joint Angular rad/s AE-I, BE 
Velocity 
HR0021 | Joint Angular mm/s” AE-I, BE 
Acceleration 
Balance & HR0022 | Hand-eye n/m BE 
Coordination coordination 
HR0023 | Ability to move | % BE 
L/R 


(continued) 
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Table 3. (continued) 


122 HR0025 | Ability to move % BE 
forward 

123 HR0026 |Standupfrom % BE 
sitting 

124 Walking HR0030 | Walking BE 

125 HR0031 |Standing up % BE 

126 HR0032 | Squat % BE 

127 HR0033 | Walking km AE-I, BE 
distance 

128 HR0034 | Walking speed | km/h AE-I, BE 

129 HR0035 | Power Assist % BE 

130 HR0036 |N of steps Number AE-I, BE 

count# 


Table 4. Clinical characteristics 


Participant | Age Years |Gender | Neurological Injury Level | Weight (kg) | BMI 
A 19 M C6-C7 53 16.0 
B 24 M C5-C6 70 23.4 
C 29 M C7 83 25.3 
D 46 M T3 75 24.5 
E 25 M C6 73 22.5 
F 23 M C6-C7 56 17.9 
G 42 M C6-C7 72 22.0 


4 Discussion 


Upon the onset of an illness or injury requiring medical treatment, patients are usually 
admitted to a hospital facility to receive clinical care. Herein, patients undergo appro- 
priate medical and or surgical treatment for their illnesses. The primary focus is strictly 
medical stabilization and health maintenance. During this inpatient hospital phase-- 
toward a path of recovery--medical professionals of varying disciplines manage and 
provide essential care for patients, including physicians and allied health professionals. 

Under appropriate settings, a medically prescribed rehabilitation course may also be 
indicated in many situations, especially following the onset of a disabling injury. The 
goal from a rehabilitation perspective is to enhance patients' functioning ability, thereby 
ultimately ensuring a safe and stable transition phase in their recovery toward community 
re-integration. This is achieved through a prescribed rehabilitation program consisting 
of conventional occupational (OT) and physiotherapies (PT). Depending on the facility, 
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this program may include a specific training period focusing on activities of daily living 
(ADL) within a simulated home and community environment before discharge. 

This simulated home and community environment—a data-driven smart medical 
living lab center housed at the National Rehabilitation Center of Korea—serves as the 
fulcrum. The linkage of pilot services’ multi-ministerial rehabilitation program will be 
linked to the disabled population throughout the local communities. Immediately after 
discharge from the hospital, patients will transition to a “smart medical healthy” living 
lab environment before returning to their respective domiciles in the community. Patients 
will spend several days in a simulated home and community setting to undergo training 
in various activities of daily living. The medical-based program focuses on rehabilitation 
exercise and sports activities specifically designed to aid these individuals in developing 
a habit of management in self-directed care and health maintenance. Unlike the inpatient 
acute hospital course, the setting herein does not require licensed clinicians to provide 
these services. And thus, participating individuals will be instructed and trained by 
non-clinical professionals with expertise in disability sports and physical education to 
ultimately ensure safe functioning status toward a stable community re-integration. 


4.1 Linkage of Multi-ministerial Rehabilitation Platforms 


The testbed participants will be recruited from the NRC inpatient population as part of 
their discharge planning from acute inpatient rehabilitation and individuals with chronic 
disabilities from the local community. The collected hospital data will be merged with 
newly acquired information under the tutelage of the four governmental ministries to 
accumulate a data bank using the onsite server. 

A living lab testbed built and operated by the NRC is currently in the development and 
establishment phase. It is expected to open and get underway by recruiting participants 
from the National Rehabilitation Center’s inpatient population and local communities. A 
smart health medical rehabilitation-based and data-driven living lab will render the pilot 
service R&D efforts designed to provide rehabilitation and physical education programs 
for individuals with a disability in the local community. 


4.2 Effects of a Collaborative Research & Development 


The overarching benefits of this integrative research and development (R&D) are multi- 
fold. The many advantages are realistic and expected to infiltrate many sectors of our 
society. 

Medical benefits will improve healthcare services through an integrated rehabilita- 
tion model with increased therapeutic efficacy expected to result via the rehabilitation 
exercise guidelines applied to various disease models. 

The benefits from a ‘technology’ perspective are also multi-fold. We expect that 
there will be further advanced development of technology for enhanced data acquisition 
that will trigger commercialization with the expansion of various contents applicable to 
other research and development areas. 

The economy can also benefit by an overall reduction in the cost of healthcare 
for all ages and social classes, especially for the elderly and marginalized population. 
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This will likely be achieved by providing welfare services for the financially destitute 
and socially disadvantaged. Industries can also work hand-in-hand to benefit from the 
expansion of the technology development into overseas markets, fueling the creation of 
job opportunities for the unemployed. 

Social advantages include reduced healthcare costs and mortality by promoting a 
healthy lifestyle and self-directed care in health management. This will mainly apply to 
the general population as people are aging and living with more chronic disabilities. The 
creation of more social platforms is also foreseeable in advantageous ways yielding an 
artificial intelligence and smart device-driven society. 


4.3 Pilot Study of Para-Athletes 


Our initial testbed run yielded results comparable to output under the circumstances 
from individuals with functioning disabilities stemming from spinal cord injury. The 
acquired data were not statistically analyzed in any specific manner or for comparison 
purposes. The raw data are provided to convey the intent of pursuing the upcoming 
complete study applying “Clinical characteristics” (Personal information; Table 4) and 
“Functioning status” (6-MWT; Table 4A) from the proposed data set. Our goal is to 
substantiate the effects of the smart-medical living lab simulated herein, which will 
apply to and ultimately lead to a governmental establishment of these fitness program 
centers specifically designed for individuals living with disabilities nationwide. 


5 Conclusion 


Patients with disabling conditions due to an acute onset of illness or devastating injury 
are discharged without proper training and education for safe return to and participation 
in their communities. They require a systematic medical-based and rehab-focused sports 
and exercise program accessible within their respective communities. A smart medical 
rehabilitation-based and data-driven living lab center can provide the necessary platform 
for successful community re-integration. 
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Abstract. Building Energy Management (BEM) and monitoring sys- 
tems should not only consider HVAC systems and building physics but 
also human behaviors. These systems could provide information and 
advice to occupants about the significance of their practices with regard 
to the current state of a dwelling. It is also possible to provide services 
such as assistance to the elderly, comfort and health monitoring. For this, 
an intelligent building must know the daily activities of its residents and 
the algorithms of the smart environment must track and recognize the 
activities that the occupants normally perform as part of their daily 
routine. In the literature, deep learning is one of effective supervised 
learning model and cost-efficient for real-time HAR, but it still strug- 
gles with the quality of training data (missing values in time series and 
non-annotated event), the variability of data, the data segmentation and 
the ontology of activities. In this work, recent research works, existing 
algorithms and related challenges in this field are firstly highlighted. 
Then, new research directions and solutions (performing fault detection 
and diagnosis for drift detection, multi-label classification modeling for 
multi-occupant classification, new indicators for training data quality, 
new metrics weighted by the number of representations in dataset to 
handle the issue of missing data and finally language processing for com- 
plex activity recognition) are suggested to solve them respectively and 
to improve this field. 


Keywords: Real-time human activity recognition - Deep learning - 
Smart home - Sensors - Embedded equipments 


1 Introduction 


Smart home services are an important provider of systems of technology. They 
focus on constructing, operating and maintaining buildings in the most cost 
and effective efficient manner. At the most primary level, smart homes deliver 
functional building services for occupants by providing them security, air quality, 
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thermal comfort, appropriate illumination, at the environmental impact as well 
as the lowest cost and over the building life cycle [20]. Reaching this vision needs 
adding intelligence from the design phase’s beginning throughout to the end of 
the buildings useful life. They use information technology during operation to 
connect different subsystems, which operate independently, so that these systems 
can share information to optimize the performance of the building. They look 
beyond the building equipment within their four walls. They are connected to 
the smart power grid, and they interact with building operators and occupants 
to empower them with practical information and new visibility levels. 

Building system is decomposed into two parts: the physics with an important 
number of appliances and building envelope, controllers for energy management, 
as well as a human part associated to building occupants. These two parts inter- 
change: the physical part is modeled and controlled by the occupants, while 
the occupants waiting for new services from the physical part. Research works 
about buildings mostly focuses on physical part and disregards the human one. 
However, the elaborate hardware and the most advanced software in the real 
applications be nothing but transistors and wires and without the people that 
use them to work more functionally. In that sense, the occupants that run a 
smart home are a crucial component of its intelligence. Consequently, in order 
to define the state of the buildings, non-measured values should be known not 
only in the physical part (i.e. air flow) but also human (i.e. occupancy and activi- 
ties). Household occupants carry out different activities each day as part of their 
normal quotidian behavior. These include getting up, taking a shower, switching 
on lights and preparing breakfast. While such activities are common to us all, 
there will be surely differences [11]. For example, one household may use the 
shower at different times of the day, while others might prefer to take a bath. 

In the literature, an important number of HAR techniques are available. They 
have significantly improved the use of deep learning techniques, and they could 
be classified into two categories. The first one is real-time. The HAR systems 
are a reactive system that depend on their environments and are in permanent 
interaction with them. They must respond to the latter's stimuli while respecting 
certain constraints, the most important of which is the time constraint. These 
systems use real-time data (i.e. the activities are detected while they are being 
performed) and in which the data describing activities are partially observed. 
The second is offline using historical data (i.e. the activities are recognized when 
they are completed) and in which the data describing activities are completely 
observed [3]. By way of explanation, regardless of the rate of sampling, the data 
captured from uncompleted activities contain fewer details about the activities 
compared to the data captured from completed activities, and this makes real- 
time HAR a more challenging problem compared to the offline HAR. 

This work presents an overview about the latest advances in the field of real- 
time HAR in smart home to obtain a better understanding of the difficulties, 
challenges and opportunities. The main contribution is to highlight the trends 
regarding the real-time HAR task from pattern classification (Sect. 2) and to 
propose new solutions for data segmentation (Sect. 3), data variability (Sect. 4), 
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datasets pre-processing issues (Sect.5) and complex real-time HAR (Sect.6). A 
discussion and a conclusion are provided respectively in Sects. 7 and 8. 


2 Pattern Classification 


Various deep learning models have frequently been used by researchers in the 
HAR field. The learning generally involves creating a statistical or probabilistic 
activity model that is augmented with large training data. The task of the model 
is to learn and recognize patterns that differentiate various classes in the training 
data, and apply this knowledge for the classification of test data. 

The HAR task proceeds from pattern recognition, where the techniques are 
decomposed on two categories. 


2.1 Ontology Based Approaches 


Various techniques have been proposed in the literature to tackle the HAR task, 
and most of the basic ones focus on location because it is a crucial part of the 
context. They assume that the semantics of a spatial position are static, that’s 
why they focus on how to specify the semantics of a spatial position and also 
focus on how to identify the spatial position of occupants [31]. Theses techniques 
work well to some scope. However, they are unable to recognize human activities 
with enough accuracy. For example, if the occupant stays at a kitchen, she or he 
may be cooking food or putting the dishes away; the efficacy of this method is 
limited because a location offer various activities. As an example, a living room 
can support various activities such as eating, playing TV games or reading. For 
now, research works fails to handle these changes in semantics. 

Another type of ontology-based approaches is thing-based techniques that 
identify dynamically the change in activities by using remote sensors to identify 
the objects that the occupant is interacting with [28]. For example, to recognize 
an activity, [32] developed a system of semantics identification that identified the 
activity space from objects forming the immediate environment of the occupant. 
These techniques try to overcome the diversity of activities in the most compre- 
hensive way possible. However, they require a complete knowledge of the domain 
to elaborate activities models. They are not robust to handle settings change and 
uncertainty, and they have implementation problems regarding the recognition 
accuracy of the used sensors depending on their cost. In addition, they propose 
to follow expert knowledge to model activities, which is time-consuming and 
difficult to maintain in case of environment evolution. In this paper, we pro- 
pose modeling this type of activity as a multi-label classification problem. We 
could use a semantic segmentation of sensors and activities and inspire from 
techniques developed in the literature such as [23] for example. Another solution 
is to exploit CNN architectures. This allows the model to relate the possibility 
that different activities could be distinguished for the same occupant. 


128 H. Najeh et al. 


2.2 Data Driven Approaches 


Another approach to recognize human activities in real-time is to monitor the 
status of different appliances to understand the behavior of the different loads 
in the household. In the literature, several load monitoring techniques can be 
implemented. They can be divided into two categories [7,27]. The Intrusive Load 
Monitoring (ILM) is a data-collection method where the devices are installed at 
each appliance node to detect the sensor events and therefore characterize in 
detail the occupant activities using deep learning techniques, for example. 

The databases generated by these systems can be labeled manually (i.e. the 
user label the monitored appliance), or automatically (i.e. the system is trained 
with examples from distinctive appliances and then recognizes the appliance that 
is being used). Generally, manual setup ILM systems master automatic setup 
ILM systems. Result’s accuracy is the main profit of this method, but it requires 
expensive and complex installation systems. 

Non-Intrusive Load Monitoring technique (NILM) is an alternative process, 
in which one single monitoring device is installed at the main distribution board 
at the household, and an algorithm is applied to determine the state of operation 
for each individual appliance. The main advantage of NILM is the fact that only 
one single monitoring device is needed. Therefore, it lowers significantly the cost 
and the intrusion at the household level. The main inconvenience is the lower 
accuracy compared to ILM systems, in particular, those with manual labeling. 

In general, the appliance of a household can be categorized in the following 
classes: (i) finite-state appliances such as dishwashers or fridges, which have 
different states, each one had its duration (cyclic or fixed) and its own power 
demand; (ii) continuously varying appliances, such as computers, which behavior 
is not periodic and have different states; (iii) on/off appliances such as light bulbs, 
which have only two states: on and off, with a fix power demand. The duration 
depends on the user; (iv) permanent demand appliances, such as alarm-clocks, 
which are always ON with a fixed power request. 

The appliances can be identified by “event-based” techniques that detect 
the On/Off changes, or by “non-event-based” methods that detect whether an 
appliance is ON during the sampled duration. These techniques can use different 
sensor measurements and features data, such as current and voltage measure- 
ments and signal waveform. The sampling rate is an important parameter in 
the complexity level of the methods of discretion. It affects not only the type of 
feature that can be measured, but also the type of algorithm that can be used. 
A detailed review about the features and algorithms is discussed in [22]. 

A high frequency sample data rate (1 s-1 min) permits more detailed analysis 
and accuracy in the detection of appliances loads. However, the large amount 
of data needs higher quality hardware and requires a capacity of processing and 
storage (locally or in the cloud) to run the algorithms. Another important issue 
in data driven approaches is related to the quality of data. In this work, we are 
interested in dealing ambiguous outliers detections in both training and testing 
data and insufficiency of labeled data. Outliers could be detected using a multi- 
class deep autoencoding model for example. Some autoencoder variants, such as 
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a variational autoencoder, can be used to better extract high-level features for 
the estimation network. We are interested also to perform experimental results 
to prove the outstanding performance of the proposed solutions and compare 
them with state-of-the-art rival methods. 


3 Data Segmentation 


In HAR task, each sensor data need to be divided into chunks, in a windowed 
manner. On each window, the features are computed, and then are used as 
an instance for learning phase. This task is difficult since occupants perform 
activities continuously, and successive activities can not be clearly distinguished. 
This section details the most used data segmentation techniques in real-time 
HAR context. 


3.1 Time Windows 


Time windowing (TW) techniques consists on dividing the data stream into 
time segments with a regular time interval. The selection of optimal duration 
of the time interval is the biggest challenge for these techniques. In fact, if the 
time interval is very large, the events could be common to many activities and 
consequently the predominant activity in the time window will have the more 
influence in the label’s choice [4]. TW techniques are commonly used in the seg- 
mentation of sensor event for real-time activity recognition. This technique is 
more favorable to the sensor time series with regular or continuous time sam- 
pling. However, in the smart home context, sensor data are often generated in a 
discrete form along the timeline where a fixed size time window is not suitable. 


3.2 Sensor Events Window 


Sensor events windowing technique consists on dividing the whole sequence into 
a set of sliding windows with an equal number of sensor events [16]. Each window 
is labeled with the last event’s label in the window, and the sensor events that 
precede the last event in the window define the last event’s context. 

It is easier to implement; however, the challenge consists on the fact that the 
actual occurrence of the activities is not intuitively reflected [18]. In fact, one 
sliding window could cover two or more activities, or sensor events belonging 
to one activity can be bust into several sliding windows. Furthermore, if several 
activities occur simultaneously, this technique is unable to segment sensor events. 
In addition, this window type differs in terms of duration, and it is consequently 
impossible to interpret the time between events. 

To solve this problem, a method that segment the sensor events into portions 
that are consistent with the incidence of each activity is proposed by [15]. This 
approach can correctly outline the extremities of the activity, but it could take 
a longer time until sufficient information to define one segmentation is received. 
Moreover, the question that arise is how to determine if two sequential sensor 
events belong to similar activities or not? This is a challenge. 
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3.3 Dynamic Window 


Unlike the previous techniques, dynamic windowing uses a non-fixed window 
size. In the literature, [26] proposed a dynamic segmentation model where the 
window is reduced and expanded based on the use of the current state of activity 
recognition, the sensor data and the temporal activity information. It is a two- 
stage methodology [21]. An offline phase consists in splitting the data stream 
into events windows, then extracting the “best-fit sensor group" from the event 
window. In the real-time phase, the dataset is streamed to the classification 
algorithm. When the “best-fit sensor group" is identified in the stream, the 
corresponding label is associated with the given segment's input by the classifier. 
A non properly annotated dataset is the most challenging for the use of the 
technique. Also, it is not able to handle complex HAR in real-time [1]. 


3.4 Fuzzy Time Window 


Another important type of data segmentation is fuzzy time windowing (FTW) 
[14]. The objective of this window's type is the generation of features for each 
sensor time series according to its evolution for a given interval of time. It is 
created for encoding multi-varied sequences of binary sensors. 

To predict the right activity label, one should not rely solely on the past 
because in some cases, delaying the recognition time allows a better decision 
to be made. Consider the following example, if a binary door contact sensor 
is activated, the activation can be associated with the following activity "the 
resident has left the house". However, it can happen that the inhabitant only 
opens the front door to speak to another person at the entrance of the house 
and returns home without going out. To improve the precision, the use of the 
activation of the following sensors is a good alternative, and it will be useful 
for example to introduce a delay in the decision-making. The longer the delay, 
the greater the accuracy. However, a problem may arise if this delay is too long 
and, in effect, the delay prevents real time recognition. While a long delay may 
be acceptable for some types of activities, others require very short decision 
time in the event of an emergency, such as a resident falling. Furthermore, this 
method is applicable only to binary sensors! data, although this is not always 
the case because a smart building also contains non-binary sensors such as CO» 
concentrations sensor. 


3.5 Outline 


Many applications implement real-time activity recognition using wearable sen- 
sors such as accelerometers. HAR using these devices is less challenging because 
the data are generated continuously with fixed frequency, allowing data to be 
segmented based on the number of sensor signals or time intervals, and they are 
limited only sample on simple activities such as walking, cycling. In contrast, 
smart home ambient sensors are normally generating data in a discrete manner; 
so, this still remains a challenge for dynamic segmentation. Besides, the classified 
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activities are often complicated, composed of a number of sub-activities where 
the duration of sensor segmentation and the boundary are difficult to determine. 

In this work, we aim to develop a new dynamic real-time sensors’ event seg- 
mentation approach to sensor data analysis which incorporates two components: 
sensor correlation computation, and time correlation calculations. 


4 Data Variability 


In the case of HAR, most variations occur in settings and data. The following 
subsections detail the variability in settings and data issues. 


4.1 Variability of Settings 


Involving inhabitants from residential buildings is still a key challenge because 
each smart home is unique due to its architecture, configuration of sensors and 
equipment, but mostly because of its inhabitants’ different profiles. The site 
models, the sensor locations and human activities/preferences are mostly not 
available, however, these are required for mass deployment solutions [27]. 

Some may be small, such as a single apartment, where the sensors may be 
fewer and have more overlap and noisy footage. Others may have multiple rooms 
and contain many sensors and devices. Indeed, the performance of an activity 
recognition system can be influenced by the number, type and location of smart 
home sensors. Therefore, a model optimized in one house may not work well in 
another because of this difference in architecture. 

In this work, we are interested to test the transfer learning methods for solv- 
ing this problem [10]. These techniques allow the use of pre-trained deep learning 
models with different data distributions. We are also motivated to analyze the 
different types of knowledge that can be updated with deep learning algorithms 
and taking advantage of recent advances in transfer learning for deep learning. 


4.2 Temporal Drift 


Thanks to the large number of sensors as well as the interaction between the 
occupants in a smart home, the initial training data constitutes a mirror of 
the activities carried out at the time of registration. A model is generated and 
trained using this data. 

Habits and behavior of occupants may change over time. The data collected 
in real time is no longer the same as the data used in the training phase. This 
is translated by the concept of temporal drift as presented in [24], and means 
a change in the distribution between the training data and the test data. Let’s 
consider the following example. A primary approach, which is prevalent in many 
buildings, is to use passive infrared (PIR) sensors for activity estimation. How- 
ever, motion detectors fail to detect activity when the occupants remain rel- 
atively still, which is quite common during activities like regular desk work. 
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Furthermore, drifts of warm or cold air on objects can be interpreted as motion, 
leading to false positive detection. 

The statistical properties of a variable that the model attempts to predict 
change over time in unexpected ways. Taking this drift into account requires real- 
time adaptation to changes in human activities using new HAR algorithm data 
in smart homes [19]. Based on real-time monitoring of human activities, we are 
interest to perform Fault Detection and Diagnostics (FDD) but also predictive 
maintenance. It is indeed possible by artificial intelligence algorithms to detect 
drifts and intervene before it is too late. Numerous research studies still make it 
possible to develop these techniques to make them truly operational, in particular 
to adapt easily to changes in piloting mode and to different configurations of the 
building [29]. 


5 Datasets Pre-processing 


Following a data collection, the performance of activity recognition is evaluated 
with datasets generated through different sensors. 

In a dataset, most of the activation data of sensors are not annotated because 
the dataset has been built with predefined activity sets to be labeled. Conse- 
quently, the activation of sensors linked to other activities are annotated under 
the “Other” label. This special class represents more than 50% of the datasets 
studied, provided by the Center for Advanced Studies in Adaptive Systems 
(CASAS) [8]. Event sequences of this class can be similar to event sequences 
of normal classes, as they can be quite different from each other. The difficulty 
of classification increases in proportion to this growth of the “Other” class. This 
leads to class confusion. In addition, each sensor activation gives only some infor- 
mation about the current activity. For instance, the activation of the kitchen 
motion sensor could indicate different activities like cooking, housekeeping or 
washing dishes. Therefore, the information offered by this sensor is exploitable 
only in coincidence with neighboring sensor activation. Thus, the time series is 
irregular and sparse, and it contains highly variable data and unbalanced classes. 

Labeling is also challenging. In fact, in supervised learning techniques, which 
are extensively used in a lot of HAR applications, the problem of the required 
target arises in the determination of the activities of occupants i.e. the labeling 
issue is usually addresses using video cameras. In order to accept the privacy of 
occupants, the use of cameras is generally not acceptable in many places. 

Also, the labeling is time-consuming. In general, the datasets for an actual 
building are labeled by the residents themselves using a graphical user interface 
and then used post-processed for the validation of research works. Missing labels 
are the most challenging problem in datasets. Let’s consider the example of 
CASAS dataset developed by Washington State University, the activation of 
sensors may not be correctly ordered in a chronological order of timestamps. 
Also, duplication of days and events (i.e. same activity label, timestamp, sensor 
and value) could arise in a dataset and a cleaning step should be considered 
before the training process. Figure 1 presents an extract from the Milan dataset 
where labels are missing. 
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Date Time Sensor ID Value Label 

2010-01-05 )8:25:37.000026 M003 OFF 

2010-01-05 08:25:45.000001 M004 ON Read begin 
2010-01-05 )8:35:09.000069 M004 ON 

2010-01-05 08:35:12.000054 M027 ON 

2010-01-05 )8:35:13.000032 M004 OFF (Read should end) 
2010-01-05 08:35:18.000020 M027 OFF 

2010-01-05 08:35:18.000064 M027 ON 

2010-01-05 )8:35:24.000088 M003 ON 

2010-01-05 08:35:26.000002 M012 ON (Kitchen Activity should begin) 
2010-01-05 08:35:27.000020 M023 ON 

2010-01-05 08:45:22.000014 M015 OFF 

2010-01-05 08:45:24.000037 M012 ON Kitchen Activity end 
2010-01-05 08:45:26.000056 M023 OFF 


Fig. 1. CASAS dataset anomaly 


The contribution in this work is to take into account the annotation of events 
in the judge of performance using evaluation metrics weighted by the number of 
representations in a dataset, F1 weighted score or balanced accuracy when the 
dataset classes are unbalanced [12]. 


6 Complex Human Activities 


An activity consists of a pattern of multiple actions over time. Typical examples 
include reading, coffee time and cooking. The occupant’s activities could be 
classified into two categories. Simple activity refers to primitives that realize 
a simple purpose or function, while complex activity consists of the temporal 
combinations of multiple simple activities over time. 

In the literature, an important number of research works have been developed 
in the framework of real-time HAR. However, they focused mostly on simplified 
scenarios involving simple activity recognition and a single occupant, and there 
is no much attention on the real-time complex HAR. 

In a house setting, occupants perform frequently different actions simulta- 
neously in a variety of temporal combinations. That’s why, activities are much 
more complex than actions, but they represent more the real life of residents. 

Occupant activities are frequently carried out in a complex manner. Activ- 
ities can be carried out in an interleaved or concurrent manner. An occupant 
may alternately wash dishes and cook, or listen to music and cook at the same 
time, but could just as easily wash dishes and cook, alternately, while listening 
to music. The possibilities are limitless in terms of activity planing. However, 
some activities could not appear in the dataset and could be also abnormal, 
such as cooking while the individual sleeps in his room. Modeling this type of 
activity is a challenge and only few research works are presented in the scientific 
literature. However, it could be modeled as a problem of multi-label classifica- 
tion. For example, in [23] a semantic segmentation of sensors and activities has 
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been investigated and this allows the model to relate the possibility that certain 
activities may or may not occur for the same resident simultaneously. 

Multi-user activity is another challenge. Besides, while the monitoring of a 
daily single resident activities is already a hard task. The complexity increases 
more with many residents. The same activities become more difficult to recognize 
for the following reasons. Firstly, in a group, a resident may interact to do 
common activities. In this case, the sensor activation reflects the same activity 
for each resident in the group. Secondly, everyone can perform simultaneously 
different activities. This produces a concurrent sensors’ activation for different 
activities. These activations are then merged in the activity sequences, where 
an activity performed by one occupant is a noise for the activities of another 
occupant. Some researchers are interested in this problem [17]. However, most 
of the existing methods are of the offline type and there is no interest to date 
for the recognition of complex activities in real time. 

The real-time activities that a smart homes want to recognize can be on the 
contrary seen as sequences of micro actions. These sequences generally follow a 
certain pattern, but there are no strict constraints on their compositions or the 
order of micro actions. To solve this problem, an important number of research 
works are present. A two-layer based LSTM technique is investigated to address 
the diversified composition of actions for HAR based on wearable sensors in [30] 
and based on videos in [9]. Another idea consists by exploiting the techniques 
developed by the natural language processing field, where the word’s context 
vary, the words vary and the texts have a multilevel hierarchical structure. 

Embedding techniques, such as BERT [5] and ELMo [6] have been investi- 
gated to operate sequential data. These techniques help the processing of long 
sequences. However, all these techniques are only experimenting with offline 
HAR. The study presented by [5] is limited by a certain size of possible sensor 
activations and vocabulary, and it is consequently impossible to obtain a repre- 
sentation of sensor values that have never been observed. In this work, we are 
motivated to investigate methods that could capture more semantics in real- 
time, i.e. that could split words and represent sensor activation into sub-words 
such as byte pair encoding (BPE) [25]. 


7 Discussion 


In this survey, we have highlighted the key challenges in real-time HAR in smart 
buildings. A taxonomy of the main components of a real-time HAR, algorithm 
(time series analysis, data segmentation, data variability, data pre-processing 
and classification) has been introduced. 

The data variability is still an unsolved problem because sensor data are very 
sensitive to the localization of sensors as well as the house configuration. We are 
interested in (i) performing Fault Detection and Diagnostics (FDD) to detect 
drifts using artificial intelligence algorithms to solve the temporal drift, and (ii) 
the use of transfer learning algorithms to solve the variability of settings. 
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Multi-occupant and concurrent activities’ recognition are the most challenged 
problems in human activity recognition task using ambient sensors. We are moti- 
vated to work on this issue by modeling this type of activity as a multi-label 
classification problem or using CNN architectures. 

Taking into account the quality of the training data was a problem rarely 
discussed in the literature. In fact, as the available information changes over 
time, the structure of the training data should also be readjusted to deal with 
such dynamic aspects. In [13], the authors have evaluated big data quality using 
different indicators: precision, accuracy, completeness, volume, timeliness and 
consistency. In this work, we are motivated by using other indicators to test the 
data quality, such as spread rate technique proposed in [2] which considers the 
global space of the data and does not look at each class alone. 

The unavailability of sensor data over long periods of time, missing data as 
well as non-annotated events are also challenging ones. In this direction, we are 
interested to (i) take into account the missing data by adding a penalization 
function in the optimization algorithm when the data are not available, and (ii) 
to judge the performance of algorithms using evaluation metrics weighted by the 
number of representations in a dataset, F1 weighted score or balanced accuracy 
when the events are not annotated. 

Finally, pattern recognition analysis and feature extraction challenges are 
solved by deep learning algorithms, such as RNN, CNN, LSTM, or hybrid archi- 
tectures (RNN and LSTM for example). Both approaches based on CNN and 
LSTM would give equivalent performance levels, but CNNs are faster in the 
training phase and therefore more suitable for real-time HAR. However, chal- 
lenges related to sequence analysis still remain largely unresolved. We argue that 
the application of language processing techniques can bring advances in solving 
some of these challenges as they deploy sequence analysis methods. In this con- 
text, our contribution to investigate byte pair encoding (BPE) methods [25] 
able to split words and represent sensor activation into sub-words and therefore 
capture more semantics in real-time. 


8 Conclusion 


This work provides an insight of new challenges for real-time HAR in smart build- 
ings. Based on the existing methods and applications, different trends are ana- 
lyzed and deals with pattern classification, data segmentation, data variability 
data sets pre-processing and complex human activities. Under these issues, var- 
ious opportunities and solutions are analyzed along with its recent application- 
based approaches. Based on the observations, the findings of the survey are 
summarized to improve the field of real-time HAR. In this work point of view, it 
is recommended to improve deep learning techniques by developing neural net- 
work architectures that takes into account the quality of training data (missing 
values in time series and non-annotated event), the variability of data, the data 
segmentation and ontology of activities for better performance improvement. 

Future work will be around the analysis of different software/hardware archi- 
tectures for real-time HAR in smart buildings. 
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Abstract. The COVID-19 pandemic has flooded a vast amount of information 
into the world. To help control this situation, good utilization of the overflow in 
data is required. However, data come in different forms, posing numerous chal- 
lenges in subsequent processing. Therefore, a uniform knowledge representation 
of COVID-19 information is needed, and ontology can play a role. The ontology 
will model patient healthcare-related data, ranging from symptoms to side effects 
and medical conditions, and the necessary precautions, especially for healthcare 
workers, to obtain protection from the COVID-19 virus. We followed Sanchez’s 
methodology to build the vocabularies, which include current ontology concepts, 
W3C standards RDF, OWL and SWRL. This work shows promising results that 
can be applied by different organizations. 


Keywords: Design ontology - COVID-19 - Healthcare - Knowledge uniform 
methodology - Knowledge representation collaboration 


1 Introduction 


The World Health Organization (WHO) [1] defines COVID-19 as an infectious disease 
that emerged as a new strain of coronavirus. WHO [1] adds that approximately 405 
million people were infected with this virus, killing almost six million from the beginning 
of the pandemic till February 2022. The scientific community is being tasked with 
controlling the spread of COVID-19, and this paper is a contributor. We aim to provide 
a knowledge representation ontology for the collection and analysis of data related to 
COVID-19 to track relevant data and eventually help control this pandemic. 

To respond to this health emergency, information-sharing across different platforms 
and systems is demanded. However, data can be heterogeneous. Systems collect infor- 
mation based on discipline-specific terminologies, which can restrain them from sharing 
the information between platforms. Hence, ontologies can offer uniform ways of repre- 
senting knowledge. Data publishers and collectors can use a shared vocabulary to collect 
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COVID-19 data. An ontology, as presented by Tudorache et al. [2], defines a common 
jargon for sharing data amongst researchers and incorporates machine-interpretable def- 
initions of essential concepts and relationships in this field. Knowledge represents the 
foundation of creating an artificial intelligent model. Dou et al. [29] state how research 
has shown that the frequent integration of semantics presents improved outcomes in data 
mining and deep learning. 

The ontology presented in this paper provides a knowledge representation model 
of COVID-19 from a healthcare perspective, demonstrating every patient’s case and 
recommending how people, especially healthcare workers, can protect themselves and 
follow appropriate safety and protection measures to decrease the risk of contracting the 
virus. 

This article is organized as follows. Section 2 presents an overview of the existing 
COVID-19 ontologies. We classify our ontology in Sect. 3, and then we build a COVID- 
19 ontology and describe how it differs from other ontologies in Sect. 4. Section 5 
specifies a few reasoning rules applied in this ontology. Finally, conclusions and future 
works are given in Sect. 6. 


2 Literature Review 


This global pandemic has given rise to several COVID-19-related ontologies in order 
to cope and take control. Each ontology models this virus from a different perspective. 
This section discusses some of these ontologies, shedding light on how our ontology 
differs. 

The COVID-19 Surveillance Ontology, presented by de Lusignan et al. [4], is 
intended to help surveillance in primary care. The fundamental objective of this ontology 
is to monitor COVID- 19 cases and related respiratory conditions using information from 
various clinical record frameworks. It was built as a taxonomy with classes including 
exposure to COVID-19, knowledge related to COVID-19, definite and indefinite con- 
traction of COVID-19. However, this ontology does not comprise any property, hence 
lessening its semantic expressivity. 

Another ontology is CODO, presented by Dutta and DeBellis [5], which is the most 
similar to our work. This ontology was designed for cases and patient information rep- 
resentation to help in publishing COVID-19 data using Findability, Accessibility, Inter- 
operability, and Reuse (FAIR) standards. It was built to facilitate the organization and 
illustration of daily-produced COVID-19 data, the relationships between the datasets, 
and the surrounding factors, for the further analysis of data. Our work differs in that it 
also covers data related to the safety and protection measures that should be applied. 

Many ontologies cover the medical perspective of COVID-19. He et al. [3] presented 
the well-known CIDO, which is a disease ontology that presents the etiology, transmis- 
sion, pathogenesis, diagnosis, prevention and treatment of this virus. The information 
includes the nature of the virus, means of transmission, common symptoms, and medical 
treatments. In contrast, our approach looks to this disease from a healthcare perspective. 
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3 Classification of Ontologies 


Ontologies can be classified based on content, goals, application technique and timing, 
domain representation, reusability, and field of application. Many ontology classification 
methods have been defined over the decades that vary in their perspectives. Ajami and 
Mcheick [7] expanded on OntoCube and added more performance standards, including 
machine-readability, i.e., whether the ontology could be easily understood and processed 
by the computer; reusability of concepts and classes to accomplish an objective; and com- 
plexity, i.e., measures of time and resources needed to achieve a certain task. Applying 
their classification technique, our ontology is considered formal as it employs the Web 
Ontology Language (OWL). Also, it is domain-specific since it describes terminology 
in healthcare, particularly COVID-19, allowing it to be semi-reusable. Finally, ours is a 
heavyweight ontology containing classes and relations, in addition to axioms and rules. 


4 Designing a COVID-19 Ontology 


Studer et al. [6] define an ontology as a "formal, explicit specification of a shared 
conceptualization" that is a clearly defined, simple, machine-readable interpretation of 
real-world concepts and their interrelationships, providing shared knowledge for the 
target community. In this paper, we adopt this definition of ontology. 

This section provides a description of the design and development methodology of 
COVID-19 ontology. Well-known methods of designing an ontology include METHON- 
TOLOGY [8], TOVE [9], Cyc 101 [10] and YAMO [11]. We follow Sánchez's method- 
ology [25] for building a medical ontology, also used by Ajami and Mcheick [7], since 
our ontology focuses on the medical field of the COVID-19 disease. This methodology 
combines both METHONTOLOGY [8] and Cyc 101 [10] and consists of five main steps 
that we follow to build our ontology, that are as follows: determine the domain scope, 
reuse the ontology, develop the conceptual model, implement the ontology, and evaluate 
it. 


4.1 Determine the Domain and Scope of the Ontology 


The domain and scope refer to the main field this ontology covers, putting bound- 
aries around the conceptualization, and the ontology's purpose. According to Ajami and 
Mcheick [7], researchers pose a set of questions to determine the domain of the ontology, 
which we answer: 


e What is the domain that the ontology will cover? 
The COVID-19 is the main domain this ontology covers. 
e What is the purpose of this ontology? 

The purpose of this ontology is to facilitate the gathering and publication 
of COVID-19-related data as semantic services. Our ontology tracks COVID-19 
patient's medical status and predicts their severity level for a better understanding 
of the nature of the virus and how patients could be treated. It also tries to reduce the 
risk of contracting COVID-19 by tracking the essential safety and control measures 
taken by people. 
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e Who will use the ontology? 

This ontology can be used by organizations willing to collect COVID-19-related 
data to help control this pandemic and know more about this disease, such as hospitals, 
government agencies, health organizations and researchers. 

e What types of questions should the information in the ontology answer? 


1) Whatare the most common symptoms of patients with COVID-19? 

2) How severe is a patient's case? 

3) Are the safety measures effective in protecting against contracting COVID-19? 
4) Is there a relationship between COVID-19 and a certain disease? 


4.2 Reuse the Ontology 


To the best of our knowledge, the ontological representation of COVID-19 patient clinical 
healthcare and safety measures is poor since many current COVID- 19—related ontologies 
tackle the disease from a medical standpoint. However, we do integrate some medical 
terms that are used to represent health data. In our ontology, we integrate concepts from: 
Schema.org, Friend of a Friend (FOAF), and SNOMED CT. 


4.3 Develop a Conceptual Model 


The following is a set of guidelines for the development of a conceptual model, as 
suggested by [7]. 


Enumerate key terms in the ontology. 

The crucial terms that describe a context need to be defined. These terms include 
nouns that represent a specific concept (e.g., a patient is described by the noun “Pa- 
tient"), attributes that describe the type and value of what is being modelled (e.g., the 
value of temperature is a float), verbs that describe the relationships between nouns or 
between nouns and attributes (e.g., a patient “is a” Person). Since standard terminol- 
ogy shall be used t model medical terms, we used SNOMED CT ontology in building 
our ontology to model the concepts of drugs and symptoms. 

e Define classes and class hierarchy. 
This step starts by defining the classes used in the ontology, then defining the 
taxonomy of these classes by matching subclasses to classes. 
e Define class properties. 

The two main class properties are object and datatype properties. These are used to 
model the relationships among different elements of the ontology, as classes alone do 
not provide enough information to represent the context behind this ontology. Object 
properties build relationships between classes by specifying the class domain of the 
relationship and its class range. The datatype property models the value and type of 
the concept, such as string, integer, and boolean. 

Define the facet of slots. 

According to [7], a slot shall be assigned different kinds of facets that frame its 
value type, allowed values and cardinalities, to be added as required. They are mostly 
represented as string, integer, and float in our ontology. 
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e Create instances. 

Individuals of a certain class are created by choosing a class, then filling the value 

slots. An example is creating Steve as an instance of Patient. 
e Develop our ontology domain. 

We model our COVID-19 medical ontology with respect to the information and 
guidelines provided by WHO. It contains information related to a patient with COVID- 
19, including symptoms and treatment, the patient’s medical history and the safety 
measures to control this virus by healthy people, patients, or healthcare workers. Our 
ontology consists of four main sub-ontologies, as presented in Fig. 1. 


In developing our ontology, we used OWL-DL, a descriptive logic ontology lan- 
guage. The Protégé ontology editor developed at Stanford University with a Pellet 
reasoner plugin was employed. 

We divided our ontology into four sub-ontologies based on the recursive algorithm 
by Le Pham et al. [15], using Even's algorithm from Amir and Mcllraith [16]. 


\ 
" suffers from 
"as | Patient Ontology =| Disease Ontology 


follows 


Fig. 1. COVID-19 sub-ontologies 


In our ontology, “Patient” is the minimum vertex separator; hence, our ontology is 
divided based on this. The first subgraph includes patient personal information and phys- 
ical factors, in addition to the precautions to follow, while the other subgraph accounts 
for the patient's clinical status in addition to the COVID- 19 disease ontology that include 
the common symptoms and treatments of COVID-19 patients. Each sub-ontology can be 
further divided into two sub-ontologies following the same reverse algorithm suggested 
by Le Pham et al. [15]. The first subgraph can be divided through the "Person", which is 
the minimum vertex separator, thus separating our ontology into two, one dealing with 
the precautions a person should follow and the second one deals with the physical state 
of a person. The second ontology can also be divided into two sub-ontologies following 
the same methodology, with the “Patient” as a minimum vertex separator; one concerns 
the patient's clinical status and the other with the COVID-19 disease. Descriptions of 
these ontologies follow. 
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Fig. 2. Part of the patient ontology 


Patient Ontology. As shown in Fig. 2, it includes a person's personal information and 
the relevant physical vital signs that are alerted when a person contracts COVID-19, as 


suggested by [1, 12, 13]. We use SNOMED CT to represent classes such as the Patient 
and some physical attributes. 
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Fig. 3. Part of the patient's clinical status ontology 
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Patient’s Clinical Status Ontology. The clinical status ontology demonstrates a 
patient’s medical history, past diseases, current diseases, hospitalizations, examinations, 
and current medications, as illustrated in Fig. 3. According to [14], people with some 
medical conditions may have worse cases of COVID-19 and require more care and 
attention. 


Disease Ontology. The disease ontology mainly represents COVID-19, as shown in 
Fig. 4, modelling its variant type, symptoms, current stage, severity, potential risk factors, 
the location where the patient is monitored, and the patient’s medication for COVID-19 
treatment. SNOMED CT is used to model the symptoms and treatments of a patient with 
COVID-19. 
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Fig. 4. Part of the disease ontology 


Safety Measures Ontology. The precautions ontology, shown in Fig. 5, models the 
protection mechanisms needed to follow to protect oneself against contracting COVID- 
19. These precautions are for healthy people, patients and healthcare workers, especially 
those working on the frontline fighting this pandemic. All these mechanisms are modelled 
in accordance with WHO’s instructions [17, 18]. 
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Fig. 5. Part of the safety measures ontology 


4.4 Implement the Ontology 


As mentioned before, we used the Protégé tool to build our ontology. We formalized 
our ontology in OWL-DL so that it can be highly expressive, and hence, it enables us to 
apply appropriate reasoning techniques. We chose the Pellet reasoner for our ontology 
as it ensures that an ontology does not contain any contradictory facts, checks if any 
instances of a class are possible, computes the subclass relations between every named 
class to create the complete class hierarchy and computes the direct types for each of 
the individuals, as stated by Sirin et al. [24]. 


4.5 Evaluate the Ontology 


Several ontology evaluation measures have been proposed. Evaluation measures the 
quality of an ontology and if the constraints and requirements have been met. This 
section covers some of the proposed evaluation metrics and assesses our COVID-19 
ontology accordingly. 

Yu [19] suggested the following ontology evaluation criteria: 


e Consistency is achieved when the ontology's set of definitions and axioms have no 
contradictions between them, according to [27]. Running Pellet reasoner shows that 
our ontology is consistent and coherent with no conflicting knowledge. 

e Completeness occurs when the represented knowledge by the ontology covers the 
domain it represents sufficiently [27]. Our ontology is complete in terms of its purposes 
and constraints. However, COVID-19 is still a new disease that is continuously studied, 
and not enough confirmed information exists. Hence, from this aspect, our ontology 
can be considered incomplete. 

e Conciseness ensures that the ontology has no redundancy, as stated by [7]. We tried 
to minimize the number of definitions of our ontology to eliminate redundant ideas 
while representing the idea fairly. Hence, our ontology is concise. 
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Expandability measures whether the ontology can be expanded to describe further 
knowledge without affecting the current, built ontology. Since much knowledge can 
still be discovered in COVID-19 disease, we built our ontology such that the core 
concepts are not altered if new knowledge is added, much as how we added the 
vaccination part to our initial ontology after vaccination data surfaced. Hence, our 
ontology is expandable. 

Sensitiveness indicates if any changes could affect the core of the ontology. As 
mentioned previously, any alterations or addition of new concepts will not affect 
our representation as in our classes and axioms; hence, our ontology is considered 
non-sensitive. 


Ontology-Level Evaluation. Srinivasulu et al. [20] set four ontology-level metrics to 
measure the complication of an ontology’s purpose. We compare our ontology to the 
metrics against the gold metric specified in Ajami and Mcheick [7]. 


The size of vocabulary (SOV) represents the overall number of classes, properties and 
individuals in our ontology. In our case, SOV is approximately 300, which is low, thus 
indicating that our ontology is not significantly large or complex. 

The edge-node ratio (ENR) represents the ratio of the number of edges to the number 
of nodes. The ENR of our ontology is one, hence indicating that the ontology is simple 
and straight-forward. 

Tree impurity (TIP) measures the divergence of the ontology inheritance hierarchy. The 
TIP of our ontology is approximately 0.5, which means that the inheritance hierarchy 
of our ontology has not deviated significantly from the rooted tree, implying that our 
ontology is not complex. 

The entropy of ontology graph (EOG) measures the number of structural models. 
A low EOG denotes more than one structural model, thus a less difficult ontology. 
Calculated using the formula mentioned in [7], our EOG is almost one, which means 
that the class structure is fine. 


Class-Level Evaluation. Brewster et al. [21] suggested class-level metrics to evaluate 
the complexity of the ontology, and we use four for our ontology: 


The number of classes (NOC) Our NOC is 76, which is relatively good. 


e The number of properties (NOP) Our NOP is roughly 130, which indicates that the 


ontology has strong reasoning. 

The number of root classes (NORC) Our NORC is 14, indicating that our ontology is 
diverse. 

Relationship richness (RR) measures the overall number of relationships divided by 
the overall sum of numbers of subclasses and relationships. Our RR is approximately 
0.5, indicating the richness of our ontology with COVID-19-related content. 


In addition, a dataset from Carbon Health data lab [28] that included cases from 


people who contracted COVID-19 was used to validate our ontology. We were able to 
partially test our reasoning rules and have gotten promising results in terms of consistency 


150 H. Mcheick et al. 


and accuracy. For future work, we’re working on fully validating our ontology with a 
complete dataset and with different scenarios. 

Overall, the evaluations show that our ontology is adequate and can be reused or 
expanded. 


5 COVID-19 Reasoning Rules 


We use the Semantic Web Rule Language (SWRL) to formulate those rules, combined 
with an OWL knowledge base, which, according to Horrocks et al. [22], extends the 
OWL abstract syntax to a high level. 


5.1 Reasoning in Our Ontology 


First Case. Used to determine the severity level of a patient’s condition and the place 
where they shall be monitored in. The National Institutes of Health (NIH) [23] specifies 
the severity level of a patient diagnosed with COVID-19 based on symptoms, physiolog- 
ical signs, and hospitalization requirements. [23] classifies the severity level of a patient 
into these four categories: asymptotic, mild, moderate and severe. An asymptotic per- 
son doesn’t exhibit any symptoms though having a positive polymerase chain reaction 
(PCR) test. A patient with mild severity level is one that exhibits symptoms but doesn’t 
have shortness of breath, dyspnea, or abnormal chest imaging. A patient with moderate 
severity level has an oxygen saturation (SpO») percentage above or equal to 94%, while 
a severe case is when the patient's respiratory rate is above 30 breaths/min. Nonetheless, 
patients with asymptotic or mild severity level shall be monitored at home, while those 
with moderate and severe cases shall be hospitalized. 


Second Case. The second case determines whether the safety measures imposed by the 
European Centre for Disease Control [26] are being applied, such as wearing a mask, 
sanitizing, and social distancing. These rules also aid in showing the effect of these 
measures on the health and lives of people not diagnosed with COVID-19 in terms of 
contracting the virus, especially healthcare workers who need to be more cautious when 
working with COVID-19 patients. Our ontology can be used in different frameworks, 
such as an Internet-of-Things (IoT) system that detects if people are actually following 
the COVID-19 safety precautions, and evaluating the effectiveness of such precautions. 


5.2 Defined Classes 


We built axioms in OWL that define the necessary information to declare an individual 
as a member of a class. Reasoners use these axioms and build the class hierarchy in 
accordance while adding further reasoning to these individuals. Most of our axioms were 
builtinside the EquivalentTo field in Protégé. Hence, whenever an individual fulfills these 
conditions, the person is considered an instance of the equivalent class. An example in 
our case is a patient — being a patient is equivalent to being a person who has undergone 
a PCR test and received a positive result. 
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6 Conclusion 


All the data generated from the pandemic can be rendered useless if not organized and 
translated into meaningful information. An ontology is a key element to extract the 
concepts from emerging COVID-19 data to help control the pandemic. We designed 
this ontology to model concepts related to patient healthcare data and help with the 
collection and analysis of the symptoms and effects on patients with COVID-19, in 
addition to reducing the risk of contracting the virus by setting well-defined precautionary 
measures. The built ontology was compared against evaluation metrics and shown to be 
of good quality and ready to be used or expanded. With the onset of vaccinations and 
new vaccination data emerging daily, future work can include extending our ontology 
with a detailed vaccine sub-ontology to cover the assessment and side effects of each 
vaccine. Nonetheless, we are in the works of validating our ontology with a complete 
dataset that tests all the reasoning rules in to fully prove the ontology’s effectiveness in 
the healthcare and safety domain. 
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Abstract. The COVID-19 pandemic took a toll on the world's healthcare infras- 
tructure as well as its social, economic, and psychological well-being. In par- 
ticular, Italy's unexpectedly high COVID-19 case and death rate from March to 
June, 2020, captured headlines due to its speed and virulence. Many governments 
are currently implementing measures to help contain and slow down the spread 
of COVID-19. The Social Response to Covid-19 Smart Dashboard was built by 
researchers at the Metabolism of Cities Living Lab, Center for Human Dynamics 
in the Mobile Age at San Diego State University and Politecnico di Milano. This 
dashboard provides an aggregated view of what people in 10 Italian metropolitan 
cities (Milan, Venice, Turin, Bologna, Florence, Rome, Naples, Bari, Palermo, 
and Cagliari) tweet during the pandemic by monitoring social media behaviors in 
the north, center, south, and islands. Moreover, the dashboard is a geo-targeted 
search tool for Twitter messages to monitor the diffusion of information and social 
behavior changes which provides an automatic procedure to help researchers to: 
associate tweets based on geography differences, filter noises such as removing 
redundant retweets and using machine learning methods to improve precisions, 
analyze social media data from a spatiotemporal perspective, and visualize social 
media data in various aspects such as weekly trends, top urls, top retweets, top 
mentions, and top hashtags. The Social Response to Covid-19 SMART Dashboard 
provides a useful tool for policy makers, city planners, research organizations, and 
health officials to monitor real-time societal perceptions using social media. 
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1 Introduction 


Social media is more ubiquitous than ever, enabling it to be a good tool to keep connected 
during the pandemic. Using automatic data processing for Twitter messages, the Social 
Response to COVID-19 SMART (Social Media Analytic and Research Testbed) Dash- 
board helps researchers search Tweets in different cities, filter noise (such as removing 
redundant retweets and using machine learning methods to improve precision), analyze 
social media data from a spatiotemporal perspective, and visualize social media data 
in various aspects (such as weekly and monthly trends, top URLs, top retweets, top 
mentions or top hashtags). The Social Response to COVID-19 SMART Dashboard uses 
multiple data mining programs, GIS methods, and advanced geo-targeted social media 
API’s to track selected topics in space and over time. There are multiple components to 
searching, processing, and visualizing social media messages from the Twitter Standard 
Search application programming interfaces or API’s. The filtered statistics of the focus 
topics and geo-targeted cities are visually represented in the SMART Dashboard. 

The daily and almost live monitoring capability of the Dashboard has great potential 
for local, state, public health agencies, and practitioners to integrate real-time information 
to investigate large-scale disease outbreaks. For example, the Social Response to Covid- 
19 SMART Dashboard can be used to study sentiments on COVID-19 and vaccines in 
Italian cities based on new policy mandates and curfews. Because of the Dashboard’s 
unique capability to capture the temporal and spatial nature of COVID-related policies, 
behaviors, beliefs, and sentiments through Twitter content revealing various trends in 
diverse geographic areas, community leaders can use this tool to closely connect to 
their constituents and mitigate social issues before they become full-blown movements. 
Another potential use is to monitor public opinion towards crisis events such as the 
SARS COVID-19 outbreak. The Dashboard visualizes the most popular media shared 
in Twitter based on the COVID-19 pandemic in real-time. 


2 Literature Review 


To provide more background on the Dashboard, the following areas will be discussed 
in detail: the impact of COVID-19 on the 10 metropolitan cities in Italy to understand 
the geographical and temporal constraints, social media analytics to delve into their use 
cases, and SMART Dashboard 2.0 to delve into the history of the dashboard. 


2.4 Impact of COVID-19 on the 10 Italian Cities 


The COVID-19 pandemic has turned the once tourist-filled cities of Italy to ghost towns 
due to quarantine measures. The SARS-CoV-2 (severe acute respiratory syndrome coro- 
navirus 2) is caused by a coronavirus, and it presents itself with symptoms that include 
"fever or chills, cough, shortness of breath, difficulty in breathing, fatigue or tiredness, 
muscle or body aches, headaches, new loss of taste or smell, sore throat, congestion 
or runny notes, nausea or vomiting, and diarrhea" [12]. During the beginning of March 
2021, Italy came into the forefront of world health news due to its rapidly rising COVID- 
19 cases and deaths as well as for being the first country outside of Asia to have such 
high cases and deaths. 
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To provide more background, Italy’s first confirmed COVID-19 case was reported in 
the Province of Lodi, Lombardy region on February 20th, 2020 [11]. The next day, Italy 
and all of Europe the first COVID-19-related death was announced in the province of 
Padova and Veneto region [1]. Due to the increasingly older residents who have a larger 
likelihood of comorbidities in Italy, the majority are at risk for the disease [2]. 

Other Twitter dashboard studies have focused on identifying real-time Twitter trend 
analysis using big data analytics and machine learning techniques [3]. For instance, Garg 
and Kaur [4] have explained the analysis of Twitter data using components of Cloudera 
distribution of Hadoop. In fact, the study’s objective assigned polarity to each tweet. 
Map reduce and Apache SPARK frameworks were used for sentiment analysis. The 
result showed that Apache SPARK is better than MapReduce. Saad and Yang [5] have 
performed sentiment analysis of Twitter data using ordinal regression. While, Ahmed 
and Rodriguez-Diaz [6] have performed sentiment analysis on online customer reviews 
as a form of visualization. Finally, Rathod and Barot [7] researched the same field to 
predic public opinion on ongoing events by analyzing tweet sentiments using machine 
learning classifiers like SVM, Naive Bayes, logistic classifier, and KNN classifier. SVM 
was found to be the best classifier with the least mean square error for the classifications. 
Garg et al. [8] have identified the trending pattern in Twitter using SPARK. These patterns 
were obtained by collecting tweets on a real-time basis and identifying trending hashtags 
at the same time. It was implemented using a big data technology SPARK streaming. 
This type of technique can help governments or companies know about more about the 
behaviors/trends of their given campaign/program and/or brand/product awareness and 
customer needs. 

The time frame chosen to provide a proof of concept for the Dashboard is from March 
3rd to June 25th, 2020. This period is divided into Phases 0, 1, and 2. Phase O started 
when the first case was reported until before the lockdown. Phase I, or the lockdown 
phase, started on March 11th, 2020 and ended on May 4th, 2020 [9]. Phase 2 lasted from 
May 4th, 2020 until June 3rd, 2020 [2]. 

Phase I is marked by increased restrictions in Milan in response to the pandemic. 
Specifically, educational institutions, religious events, cultural centers, and all events and 
places that required gathering were prohibited [9]. This included professional sporting 
events. Visits to family and relatives were prohibited as well as patronizing bars and 
restaurants. Dining institutions were allowed takeout with limited hours. Face masks 
were required in all public spaces indoors and outdoors. In addition, there was a self- 
certification form that the government required the residents to fill out and keep on their 
person whenever they left their homes that enabled contact tracing measures [9]. The 
lockdown was exacerbated when military force was ordered to keep lockdown measures 
in place. Due to travel restrictions, no airports were open for use. The only travel of any 
kind allowed was to the grocery store, pharmacy, or the hospital. Next, Phase II marked 
the easing of restrictions in Phase I. Businesses opened without limits to their hours of 
operation [9]. Some airports opened enabling reduced international travel. Public parks 
also opened as well as public transportation with reduced capacity [9]. 
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2.2 Metropolitan Italian Cities 


Forthis study, Italian major metropolitan cities were explored to understand the intercon- 
nections between geographical location, number of COVID-19 cases, social response 
to the pandemic and locally-enforced measures based on Twitter data. Table 1 shows 
the 10 cities that were selected across Italy. These cities were chosen by the Crowdfight 
International Team, a multidisciplinary research group, based on economic and cultural 
factors. Since the outbreak started in the North, the team decided to start there while 
other cities were added over time. Milan and Venice were chosen to represent the North- 
western region. Turin and Bologna were chosen for the Northeastern region. Florence 
and Rome represented the Central region. Naples and Bari claimed the Southern region. 
Palermo and Cagliari represented the Islands. 


Table 1. Socio-demographics for the 10 Italian metropolitan cities [13] 


Type City Region Area (km?) Population | Radius Latitude | Longitude 
(NUTS-3) (n) surveyed 
(miles) 
North | Milan Lombardy | 1,575 3,190,3405 38 (24) 45.4642 | 9.1899 
West 
North | Turin Piedmont |6,829 2,293,340 | 59 (37) 45.0703 7.6868 
East Venice | Veneto 2,462 858,455 |56 (35) 45.4408 | 12.3155 
Bologna | Emilia 3,702 1,005,831 |9 (6) 44.4949 | 11.3426 
Romagna 
Center | Florence | Tuscany 3,514 1,007,435 |56 (35) | 43.7696 | 11.2558 
Rome Lazio 5,352 4,336,915 67 (42) 41.9027 | 12.4963 
South | Naples Campania | 1,171 3,128,702 45(28) | 40.8517 | 14.2681 
Bari Apulia 3,821 1,251,004 | 48 (30) | 41.1171 | 16.8719 
Islands | Cagliari | Sardinia 1,248 431,302 |37(23) | 39.2238 | 9.1217 
Palermo | Sicily 5,009 1,276,52 | 11(7) 38.1157 | 13.3615 


2.3 SMART Dashboard 


The idea of the "prototype created by the Center for Human Dynamics in the Mobile 
Age at SDSU was to facilitate the rapid dissemination of official alerts and warnings 
notifications from OES during disaster events via multiple social media channels to 
targeted demographics" [15]. The platform can identify and recruit top 1000 social 
media volunteers based on their social network influence factors and can aid government 
agencies to communicate more effectively to the public [14]. 

In our study, this same Dashboard was refitted to 10 metropolitan cities in Italy. 
More specifically, the north, center, south and island cities of Italy [14]. The backend 
was improved and mounted on larger servers. 
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3 Methodology 


3.1 


Data Collection 


To provide the analysis, the team began by collecting Twitter data through the Twitter 
Standard Search API. This involved making a Twitter Developer account, requesting 
access tokens and keys followed by authentication of said keys. The API allows for 
collecting specific metadata, so the researchers had freedom to choose which ones to 
use for the study. In addition, Table 2 shows the keywords that were used to harvest the 
Tweets. These were chosen by the Crowdfight International Team in partnership with the 
Metabolism of Cities Living Lab under the Center for Human Dynamics in the Mobile 
Age (HDMA), after discussions with Italian colleagues as well as medical professionals. 
Keywords were selected based on popularity based on hashtag and word of mouth. 


Table 2. Social response to Covid-19 smart dashboard selected keywords 


ID 


Italian keywords 


distanziamento sociale 


English keywords 


Social distancing 


Description 


Refers to the rule 
of being at least 6 
feet apart in public 
and private spaces 
to decrease the 
spread of 
COVID-19 


positivi 


Tested positive 


For COVID-19 
virus 


Stay at home 


Stay at home 


Refers to the 
measure used by 
governments to 
decrease spread of 
COVID-19 


vaccino 


Vaccine 


self-explanatory 


Coronavirus/COVID/COVID-19 


Coronavirus/COVID/COVID-19 


Caused by a 
coronavirus called 
SARS-CoV-2; the 
cause of the 
pandemic 


sintomi 


Symptoms 


Symptoms of 
COVID-19 


mascherine 


Masks 


Medical masks 
used to prevent 
spread of 
COVID-19 


(continued) 
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Table 2. (continued) 


ID | Italian keywords English keywords Description 

8 | quarantena Quarantine Measure to reduce 
spread of 
COVID-19 

9 | amuchina Hand sanitizer Hand sanitizer 


(slang); may also 
refer to a brand of 
bleach and brand of 
sanitizing products 


10 | Giuseppe Conte Giuseppe Conte Prime Minister of 
Italy (former) 


3.2 Data Collection 


In order to understand how the data is analyzed it is important to understand the client 
and server framework in Fig. 1 below. 


Clie Serv. 


nt TOP er 


gatP 
rote 
JS cted RESTful API 
Libr Topi | (NodeJS) 
ary C | CC 4] 
- ..... Web Server (NodeJS) 
Twitter Search 
HIML5/IS/CSS3 EngipedPython) 
(MongoDB) 


Fig. 1. Data framework [4] 


The server side for the Dashboard is explained below. For the database, the social 
media data tends to be more unstructured, so a NOSQL database, specifically Mon- 
goDB was used [10]. The Twitter Search Engine, coded in Python, was used to specify 
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keywords, time period, and automate collection [10]. The web server used is written 
in NodeJS so that there would not be a need to switch to other server-side languages 
to implement the server [10]. This was specifically written so that JavaScript and node 
modules can be utilized to expand the functionality. Having NodeJS for the server also 
enabled for easier REST API creation, since the API is also built with NodeJS [10]. The 
client side of the framework is built upon HTMLS (HyperText Markup Language 5), 
JavaScript (JS), and CSS3 (Cascading Style Sheets, Version 3) as the base. On top of 
which are various JavaScript libraries to be discussed in Table 3. 


Table 3. JavaScript libraries in the client side [10] 


JS library Utility 

jQuery Easily handles HTML document traversal and manipulation, event 
handling, animation and AJAX; the API can also be used across multiple 
browsers 

Bootstrap Well-known framework for HTML, CSS, and JS, creating responsive 
projects on the web 

jQuery MD5 Encrypts password in MD5 encoding format 

Leaflet Used for maps 

D3JS Used for Word Cloud section 

Dygraph Used for line chart in the Trend Section 

Morris.js Used for bar chart in the Trend and Word Cloud Sections 

dataTable Displays results in table format with filter and sort functions 

Moment.js Formats date and time 


Twitter widgets | Permits Tweets to be displayed in Twitter style; used in Top Retweets 


OWL Carousel Displays images in a carousel format; used in Top Media 


3.3 Dashboard Features 


Due to the flexibility of the original SMART Dashboard 2.0, the Social Response to 
COVID-19 Dashboard was created by first changing the geo-tagged tweets during data 
collection then changing the keywords and filtering out specific links that may be deemed 
inappropriate or unrelated to the cause on the SMART Dashboard. Each section of the 
COVID Dashboard is discussed below. 

The first few components that the user sees is the screen in Fig. 2 below, containing 
the Dashboard Toolbar on the far left, the SMART index at the top, and the Trend and 
Top Media sections below the SMART index. It also houses the “Stop Auto Refresh" 
button in order to enable researchers to stop the feed and conduct analyses. 

Dashboard was created by first changing the geo-tagged tweets during data collec- 
tion then changing the keywords and filtering out specific links that may be deemed 
inappropriate or unrelated to the cause on the SMART Dashboard. Each section of the 
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Fig. 2. Initial interface of social response to Covid-19 smart dashboard 


COVID Dashboard is discussed below. The SMART Dashboard 2.0 Toolbar, on the 
far left, contains the shortcuts of each component on the Dashboard. It also houses the 
keywords used to extract the Tweets. In addition, it contains the “Download” button to 
gain access from the data in the dashboard, the Privacy Policy, and Feedback buttons. 
The “Home” button enables the selection of keywords and filtration of certain Tweets 
that may be inappropriate or that adds noise to the findings. The toolbar also enables the 
selection of keywords simply by checking and unchecking the keywords desired. 

The SMART Index, which consists of the four multi-colored blocks across the top, 
shows the most current metrics from the last 10 min it refreshed. The blocks will be 
discussed from left to right. The first block (blue) from top to bottom shows the number 
of Tweets harvested within the past hour, the date they were extracted, and the distribution 
of the time that each tweet was extracted. The second block (green) shows the number 
of Tweets extracted in the past 24 h, current date, and distribution of the Tweets over 
time. The third block (yellow) contains the number of Tweets since the day before the 
current date. It also contains the distribution of the number of Tweets from the day before 
and the current date. The fourth block (pink/salmon) shows the number of Tweets since 
the beginning of collection and the distribution of Tweets from the beginning of the 
Tweet harvest until the current date The Trend Section shows the frequency of Tweets 
generated by the keywords over time through a series of line graphs. Users can hover 
over any section of the graph and it will show the Tweets, both filtered and unfiltered, in 
the time frame. Any point in the line can be clicked to show the Tweets at the selected 
timeframe within the point selected. In addition, the tabs on the top can change said time 
frame. In Fig. 2, the graph shows how users can visualize Twitter metrics from the past 
10 min, hour, daily, weekly, and monthly, therefore shrinking the graph towards the left. 
The bottom sliding scale can also change the distribution of the timeline of the graph. 

The Top Media Section on the lower left of Fig. 2, shows the most shared images 
posted within the timeframe. The user has options to change the time frame, whether 
to show all media from the beginning of extraction, a week of current date, a day from 
current date, and from the current date. 

The Top URL Section shows the most posted links or web pages within the timeframe. 
The user has options to change the time frame, whether to show all URL’s from the 
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beginning of extraction, a week of current date, a day from current date, and from the 
current date. Figure 4 shows it all. A unique feature in the Social Response to COVID- 
19 Dashboard is the Word Cloud Section in Fig. 3. It includes a word cloud and most 
frequent vocabulary words table within the selected time period. The word cloud function 
contains the most frequent vocabulary words within the corpus at any chosen time period. 
The size of the words indicate a higher frequency, while words with smaller fonts are 
less frequent. Word clouds are an intuitive, decorative, and convenient way to see most 
common keywords in a corpus. Future developments for certain word clouds can include 
using stopwords, or words that are used so commonly that they provide little to no value 
to the visualization. For example, in English, this could include articles and prepositions, 
like “the” and “into.” This would naturally mean selecting a particular language, which 
when harvesting geo-tagged Tweets, do not guarantee one specific language. 
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m chi $ og si ni : ies x c 
Sintomi, ché. n ack, ni d f es cori . | 
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Pleoniere @faljziora Ü da prima 
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Fig. 3. Word cloud 


In addition, the Vocabulary Frequency Table shows the most frequent words in the 
corpus in the selected time frame. The information is presented in bar chart form arranged 
from most frequent to least frequent. Another unique feature of the Dashboard is the 
Tweets in Cities section shows the normalized tweeting rates by city population within 
the certain time period selected. Basemaps can be changed to the user’s preference. In 
our example in Fig. 4, the map is based in Milan, Italy and the selected time period 
is all Tweets since the beginning of the extraction. Other options include a week from 
current date, a day from current date, and the current date. What may also be notable for 
researchers is the geographic visualization of where the tweets were collected as well as 
the collection radius and other useful statistics like the total number of Tweets collected 
in the selected time period and the latest population information that the API can find. 

The most common Retweets from the selected time period are displayed in the Top 
Retweets section. Like the other sections, users can select which time period they want: all 
Tweets since the beginning of extraction, a week from the current date, the day before the 
current date, and the current date. Each Retweet has its frequency next to it. Retweets are 
important because they are quantifiable measures of influence. They also heavily affect 
a corpus if the study does not require original Tweets.The Top Mentions section shows 
the most frequent user references (beginning with ‘@’) in the selected time period. This 
section is notable because mentions are quantifiable measures of reference. It shows the 
frequency of interaction between the Twitter users within the collected corpus. Their 
corresponding frequency is displayed next to each user that was mentioned. Users can 
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Fig. 4. Tweets in cities 


select which time period they want: all Tweets since the beginning of extraction, a week 
from the current date, the day before the current date, and the current date. The Top 
Hashtags, which refer to an idea or theme of a tweet, are shown below the Top Mentions 
section. Users create this hashtag to refer to certain movements, using the pound sign 
(P). Like mentions, these are also quantifiable measures of reference and levels of 
interaction between users and hashtags. The corresponding frequency is displayed next 
to each frequent hashtag in the time period. Users can select which time period they 
want: all Tweets since the beginning of extraction, a week from the current date, the 
day before the current date, and the current date. The last shows the Geocode Status of 
the Tweets collected in the selected time period. This is meaningful for the researchers 
because it gives context to the successfully geocoded tweets in the corpus. It can give 
insight into error rates, so future experiment parameters can be adjusted accordingly. 
Corresponding counts and percentages are displayed next to each status (Fig. 5). 
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Fig. 5. Top hashtags 
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4 Discussion and Future Work 


This type of dashboard is successful in filtering certain websites and content and the 
unique combination of visualizations increases the potential of the tool to be used in 
many different settings. For the purposes of social response to COVID-19, it allows 
policymakers to understand the current behaviors of society and can be used to observe 
public opinion during and after crisis events or disease outbreaks. The SMART Dash- 
board is available for use to assist response and assistance efforts during the pandemic. 
Real-time public health information and major events captured using social media are 
now at the forefront of behavioral measurement, disease surveillance, health promotion, 
and more. Different cities and regions may reveal different patterns of social media mes- 
sages and trends. By analyzing the context of social media messages, linking place and 
time together we can discover more meaningful patterns and insights depending on the 
goals of the study of disease outbreaks and social media activities. Having expounded 
on the Dashboard’s capabilities, it is useful to note that the limitations of this study are 
dependent on the Twitter Standard Search API, the capacity of the server to store data, 
the extraction parameters in data collection, and the specific keywords used in the study. 
With the constant sharing of ideas online, it is impossible to capture the totality of themes 
online. In addition, certain natural language processing techniques for the word cloud 
can be improved by implementing specific stopwords in order to see specific keywords 
rather than articles and prepositions. Work can be done to make the techniques agnostic 
to language including stopword adjustments. 
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Abstract. The coronavirus known as COVID-19 is the topic of the hour 
all over the world. This virus has invaded the world with its invariants, 
which are characterized by their rapid spread. COVID-19 has impacted 
the health of people and the economy of countries. For that, laborato- 
ries, researchers, and doctors are in a race against time to find a cure 
for this pandemic. To combat this virus, cutting-edge technologies such 
as artificial intelligence, cloud computing, and big data have been put in 
place. In our work, we use Internet of Things (IoT) technology. The use 
of IoT in an efficient way can lead to detecting infected people and avoid- 
ing being contaminated. In this paper, we are interested in the remote 
medical monitoring of patients who have tested positive for COVID-19. 
We propose a meta-modeling technique to model the IoT architecture. 
Then we implement two IoT solutions that permit the remote medical 
monitoring of patients infected with COVID-19 and the respect of social 
distancing by instantiating correct models that conform to the proposed 
meta-model in order to mitigate the COVID-19 outbreak. 


Keywords: Internet of Things - Meta-model - Covid-19 - Cloud - 
Android - Sensor * Actuator * Iot application 


1 Introduction 


Since 2019, the world has been living with the pandemic of coronavirus known as 
COVID-19, and it has impacted many people around the world. COVID-19 has a 
lot of variants, such as the Delta and Omicron variants, which are characterized 
by a high propagation speed in a short period of time. Generally, the virus causes 
respiratory sickness because it attacks the lungs. COVID-19 has three categories 
of symptoms: most common symptoms such as fever, less common symptoms like 
loss of taste or smell, and severe symptoms such as difficulty breathing. After 
much research, some institutes are able to produce vaccines against COVID-19. 
They are approved by the World Health Organization (WHO). Globally, as of 
February 18th, 2022, there have been 418,50,474 confirmed cases of COVID-19, 
including 5,856,224 deaths reported to WHO. As of February 15th, 2022, a total 
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of 10,279,668,555 vaccines have been administered [25]. Despite the vaccines dis- 
covered, the battle is not over. Not only because the demand for the purchase 
of the vaccine is greater than the production and poor countries do not have 
the financial means to buy it, but also because of the mutations of the virus. 
For that, we need to reinforce protection by using emerging technologies. The 
Internet of Things can play an important role in the fight against COVID-19 
and can be considered as an enabler by providing smart and innovative solu- 
tions. Collecting the data, transmitting data, reacting after the processing of 
data, and visualizing the data are exactly what the IoT provides to us and what 
is needed to deal with the virus. IoT, can be applied in different domains like 
transportation, agriculture, and, in our case, healthcare. It helps to prevent and 
detect diseases and react at the right time. In the case of COVID-19, we can use 
an IoT-based wearable body sensor to monitor patients remotely. But, creating 
applications in this domain is too hard due to the complexity of IoT systems. In 
addition to the storage constraint, we have a huge number of connected devices, 
communication, and processing infrastructures. The design of architectural soft- 
ware systems enables architects to grasp the construction of complex software 
systems. However, their colloquial description may cause confusion, resulting in 
erroneous software system implementation. In this paper, to address this issue, 
we propose a meta-model and implement two scenarios using an IoT system to 
fight COVID-19 using some sensors and actuators in order to test the proposed 
meta-model. The first one is to monitor contaminated people and alert the con- 
cerned authorities. And the second one is to apply social distancing when people 
wait their turn in supermarkets for payment, in administration, etc. 

'The remainder of the paper is organized as follows. Section 2 outlines related 
work. Section3 presents the proposed meta-model. Section 4 introduces our use 
cases and their modeling. The necessary equipment, the implementation details, 
and results are discussed in Sect. 5. The conclusion is reported in Sect. 6. 


2 Literature Review 


The Internet of Things refers to the process of connecting the physical world to 
the Internet, including things such as light bulbs, medical devices, and traffic 
lights via a network. Process the data collected by sensors in the cloud. The 
result of the processing phase is actions in order to react and provide the user 
with useful information through applications. For this, several business sectors 
can benefit from IoT such as home automation, health, transport, industry and 
agriculture. Some researchers propose methods and techniques to assist devel- 
opers and architects from the modeling phase to the final IoT product. Previous 
works are classified into two categories: those with a high level of abstraction and 
others with a low level of abstraction. In the second category, we notice different 
application domains. In the first category, in [5], authors propose a new domain- 
specific language called BIoT. This makes the creation of software architectures 
for IoT applications easier for software developers and permits them to avoid 
errors in the design step. Kallel et al. [9] use a business process model (BPM) 
and integrate the IoT-fog architecture with the cloud. Heterogeneous and non- 
heterogeneous IoT resources, resource metrics, and QoS constraints have all been 
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gained by the architecture. Authors in [21] propose a profile called UML4IoT 
that enables us to exploit IoT in manufacturing systems. It enables the designer 
to create a cyber-physical component utilizing software and system standards. 
For the modeling and specification of design patterns, authors in [22,23] propose 
an approach to designing the SOA design patterns using the SoaML standard. 
Next, they attribute to them a formal semantic using the Event-B method. In 
past work [19], we applied our methodology of modeling IoT systems to a smart 
home system. In the other category, we find several application domains. In the 
agriculture domain, authors present an IoT architecture for smart farming [6] 
based on wireless sensor networks and a plug-and-play approach for standalone 
nodes. The used algorithm increases the network’s lifetime. In smart home field, 
paper [12] presents a smart home system that improves the quality of life of a 
home owner using IoT techniques. The authors implement a server to intercon- 
nect things in the home and a web application to control these things in real 
time. In the healthcare sector, the study [20] designed a system that detects 
the quality of sleep in a patient by collecting some data with sensors. These 
are transmitted to a local server in order to use the random forest classification 
method for predictions and classification. Authors in [13], add a new technol- 
ogy to the IoT, which is Deep Learning. An optimized neural network with an 
accuracy of 97% can be used in an IoT system to detect fall. Authors in [24], pro- 
pose an approach that combines IoT, fog, and cloud computing technologies in a 
healthcare monitoring system for healthy aging in a familiar setting to improve 
quality of life. In [15], authors predict and analyze indoor air quality, which is 
important in the case of infected people in quarantine using IoT technology and 
a machine learning model with high accuracy. Baskaran et al. [3], developed a 
system that permits the detection of COVID-19 infection in a work environ- 
ment by detecting if a person is wearing a mask and thermal image. In [8], the 
authors propose a healthcare system based on the IoT using Neural Networks. 
The system processes data and makes decisions using the fuzzy logic system. 
One of the problems caused by COVID-19 is breathing. The lack of oxygen and 
oxygen concentrators in developed areas prompted the authors in [14] to propose 
a low-cost device that is easy to build. In the public safety and environmental 
monitoring domain, papers [1,2] analyze the air quality in open areas using air 
quality sensors and air pollutant data. T'his analytic permit to classify regions if 
they are good to live in or unhealthy. In [26], the authors propose a framework 
that integrate a Fog technology. The framework minimized the delay to 896 and 
had an accuracy rate of 9596. In the transportation field, a previous study [17] 
proposed a system that would permit monitoring of the platform automatically 
with the arriving trains using IoT sensors and actuators. Finally, any IoT system 
needs to be secure (unauthorized persons). For that, authors in [10] propose an 
intrusion detection system for the smart IoT environment. A system was devel- 
oped to remotely manage IoT devices using 3G connectivity technology in [16]. 
We have a vertical layer that across the horizontal layers of IoT which is secu- 
rity layer. For that, a profile called IoTsec is discussed in [18]. This UML/SysML 
extension claims to describe IoT security knowledge, and it can be considered as 
a first step to building a robust modelling language for IoT systems in terms of 
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Table 1. Comparison between previous work 


IoT standard modeling IoT implementation 
Criteria related work Standard | Message | Connector | IoT Quality | Modeling | Implementation | Respect the IoT 
modeling | type pattern |attribute | tool architecture 
Thramboulidis et al. [21] | Yes No Yes No No No Yes - 
Kallel et al. [9] Yes No No No Yes No Yes - 
Borelli et al. [5] Yes No Yes No No No Yes - 
Baskaran et al. [3] No No No No No No Yes No 
Khan et al. [11] No No No No No No Yes No 
Bhardwaj et al. [4] No No No No No No Yes No 
Our approach Yes Yes Yes Yes Yes Yes Yes Yes 
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Fig. 1. The proposed meta-model 


security and hence user safety. In [7] authors give a survey on how to deal with 
privacy and security while using IoT applications. 

By comparing previous works, we find some gabs in the conceptual part as 
well as the implementation part. However, most of the previous studies do not 
take into account how the components of an IoT system are interconnected in 
their models, as well as the nature of the data exchanged in the IoT network. 
Furthermore, some meta-models or profiles are specific to a particular domain 
application. For the proposed IoT applications, the major drawback of some 
applications is that they do not respect the general architecture of the IoT. 
In addition, they don't provide the system modeling, which is an important 
step before the implementation phase. Our work belongs to the two categories 
mentioned before; we describe our system with a meta-model at a high level of 
abstraction and we implement it. 


3 The Proposed Meta-model 


We propose a meta-model to describe the Internet of Things architecture as 
shown in Fig.1. The proposed meta-model takes into consideration all the 
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components that we can find in an IoT architecture, the connections between 
these components, and the type of the exchanged messages. The following are 
the fundamental components of the meta-model: 


1. IoTComponent: represents entities that construct the IoT architecture. It’s 
can be a sensor, an actuator, an IoT Gateway, an IoT Cloud Platform, or an 
End Device. 

2. PhysicalEntity: it is the object that we want to make connected or intelli- 
gent. It can be attached to a sensor or an actuator. 

3. Port: IoT Component interact with each other through ports. These ports 
are linked to one or more interfaces that are either required or provided. 
'The « «Port» stereotype extends the Port meta-class. We distinguish two 
types of ports: Service to say that the IoT Component produces a service, and 
Request to say that the IoT Component needs a service. 

4. Interface: interaction with the environment is enabled via interfaces, which 
are sites of communication. There are two sorts of interfaces for an entity. The 
Provided Interfaces list the services that the component offers. The required 
interfaces define the services that other components must supply in order for 
the component to perform properly. 

5. Connector: it guarantees that a provided port and a required port are con- 
nected. We have two types of connectors, assembly and delegation. 

6. ServiceInterfaces: is used to explicitly model the provided and required 
operations. 

7. MessageType: it specifies the data that are sent between IoT Component. 


We have added design patterns to ensure some quality attributes. These design 
patterns are represented as an interface to be implemented. For example, cer- 
tain devices can't connect directly to a network because they don't support the 
appropriate communication methods. A DEVICE GATEWAY design pattern 
proposes a layer in the IoT gateway that translates the communication technol- 
ogy to the right one. Connectivity, cost, and reusability are quality attributes 
guaranteed with this pattern. 


4 Case Study 


In this section, we present our use cases. Figure 2 shows the general structure of 
our IoT system. We are dealing with two scenarios. The first one is to capture 
the corporal temperature (the most common symptom of COVID-19) of contam- 
inated people through a sensor. Also, their heart beat and oxygen saturation. 
Next, the doctor can visualize the value of the collected data via a website or 
Android application. If the temperature is high or if there is a breathing prob- 
lem, the doctor will react immediately by generating an alert in order to avoid 
any further complications. The second scenario is to apply social distancing in 
shopping queues or offices in order to avoid the spread of COVID-19 by using 
sensors and actuators. 
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Fig. 2. General architecture of the IoT system 


4.1 Modeling the System 


Based on the meta-model and the use of the diagrams of the Unified Modeling 
Language (UML) version 2.5, it is possible to model the structural and behav- 
ioral views of systems. We use the component diagram from UML to model the 
structural features of our system. It is represented in Fig. 3. 


1. IoT Components: three sensors and one actuator. The Idoom Router plays 
the role of an IoT Gateway and Firebase as an IoT Cloud Platform. An 
Android application to visualize data. 

2. Port: sensors provide data, so they have Service port, and the Buzzer needs 
an action to react, so he has a request port. Other IoT components need and 
offer services, so they have both types of ports. 

3. Connector: sendDatal, sendAction and requestService are connectors that 
link provided and requieed interfaces. 


To model the behavioral features of our application, we use the sequence dia- 
gram of UML. Figure 4 shows the sequence diagram of the first scenario, in which 
sensors send the collected data to the IoT Cloud Platform via the home router. 
After that, the doctor subscribes to the Firebase Cloud Platform in order to 
visualize the data of patients. Firebase offers a real-time, reliable, and exten- 
sive database, provides APIs to developers (ease of integration), and allows the 
management of users who have access to the collected data and its community. 
Figure5 represents the sequence diagram of the social distancing scenario. The 
ultrasonic sensor measures the distance between two people and sends the value 
to Firebase. If the distance is less than 1.5 m, Firebase sends action (turn ON) 
to the buzzer. This last will make a sound, which means that social distancing is 
not respected. Through the Android application, we can see the status of buzzers 
(ON or OFF) and even whether social distancing is respected or not. 


5 Implementation and Results 


In this section, we present the materials used (software and hardware) in the 
development. A typical IoT architecture is composed of four layers: 


1. Physical layer: is composed of sensors and actuators. 
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Fig. 3. Structure of the IoT system 


2. Networking Layer: we find gateways to interconnect the whole system to 
Ethernet. 

3. Processing layer: represents the cloud to do the necessary processing and 
storage. 

4. Application layer: here, we find applications to visualize data and react if 
necessary. 


5.1 Hardware 


In order to develop our system, we need some materials, such as sensors and 
actuators. In our case, we need micro-controllers, three sensors, and an actuator. 
The materials that we found in the perception layer are listed below: 


— NODE-MCU: known as ESP8266, it is a low-cost card with an open source 
firmware and a small size. The NODE-MCU has a WiFi interface that is ideal 
for connected objects. 

— Sensors: 

1. MLX90614 Sensor: is a non-contact temperature sensor with a high 
accuracy, it's used for human body temperature measurement. 

2. MAX30100 Sensor: used to measure heart rate using the photoelectric 
method and oxygen saturation. 

3. Ultrasonic Sensor: it measures the sensor's distance from the obstacle. 

— Actuator: a buzzer which is an actuator used for generating alerts. 

— Cables: are used to connect sensors and actuators to the NODE-MCU. 


5.2 Software 


We have the Arduino IDE, which is used to program the ESP8266. The card 
supports many programming languages. Among them, we chose embedded C, 
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Fig. 4. The Sequence diagram of the remote healthcare monitoring 


which is an optimised language dedicated to embedded systems. We need to 
install a module in order to detect the ESP8266 through the board manager. As 
an IoT platform, we chose Firebase because it is secure with free multi-platform 
authentication, scalable API’s, and contains a real-time database. To visualize 
the data, we developed an Android application using Android Studio and the 
Java programming language. 


5.3 A Part of Code Sources 
For the embedded part, we first need to install some libraries. 


— Firebase library: to be able to connect to Firebase as mentioned in line one 
of Listing 1.1. 

— WiFi library: to use the WiFi module of the ESP8266, mentioned in line 2 of 
Listing 1.1. 

— Sensors libraries: to use functionalities offed by sensors, as mentioned in line 3 
of Listing 1.1. «Adafruit MLX90614.h> library to manipulate an MLX90614 
corporal temperature sensor. 


Listing 1.1. Import libraries and declarations 


1| #include ‘‘FirebaseESP8266.h” 

2| #include <ESP8266WiFi.h> 

3| #include <Adafruit_ MLX90614.h> 

4| #define FIREBASE_HOST ‘“‘database link" 
5 

6 

7 


#define FIREBASE_AUTH ‘‘database key” 
#define WIFLSSID ‘“Fixbox_1” 
#define WIFI PASSWORD “IoT” 
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Fig. 5. The Sequence diagram of the social distancing application 


After that, we need to share the link of the real time database as well as 
the key with certain devices (for security) in order to be able to read value 
from the database and write collected data into it. Both the database and the 
key are needed in the physical layer (ESP 8266) and the visualization layer 
(Android application). Finally, we used WiFi as a communication technology 
because it is supported by most devices. The Listing 1.1 shows the declaration 
of two constants in lines 6 and 7, which refer to the SSID and password of 
the IoT Gateway. Listing 1.2 presents two predefined methods that permit to 
capturing ambient temperature and the temperature of objects without contact. 
We calibrate the object temperature value by comparing the obtained values 
with those of a real thermometer and the sensor. 


Listing 1.2. Functions to manipulate the MLX sensor 


void loop(){ 
temp.amb = mlx.readAmbient TempC(); 
temp.obj = mlx.readObjectTempC() + calibration; } 


WN e 


The first line in Listing 1.3, is an instruction that permits storing the tem- 
perature value in the real-time database. The second line is used to read data 
from the real time database of Firebase. 


Listing 1.3. Methods to read and write data from Firebase 


void loop(){ 
Firebase.setFloat(firebaseData, ‘“‘Patient_1/Temperature”, T); 
T1 = Firebase.getFloat(fbdo, *Patient 1/Temperature"); } 


V = 


w 
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For the second scenario of social distancing, while waiting turn in line, people 
should wear a device like a medal that contains an ultrasonic sensor and a buzzer. 
First we declare the type of pin if it is used as an input (to collect data) or 
output (to react) as mentioned in the three first lines of Listing 1.4. Next, we 
calculate the distance between the two people (line 8 of Listing 1.4). After that, 
if the social distancing is not respected, the buzzer will turn ON to alert the 
person and update the state of social distancing in the application to NOT 
RESPECTED. Otherwise, the buzzer will stay off and the status of social 
distancing will be RESPECTED. 


Listing 1.4. The setup and void functions 


void setup() { 

pinMode(trigPin, OUTPUT); 
pinMode(echoPin,INPUT); 

pinMode (Buzzer, OUTPUT) } 

void loop() { 

digitalWrite(trigPin, HIGH); 

delay Microseconds(10); 

digitalWrite(trigPin, LOW); 

duration = pulseIn(echoPin, HIGH); 

distance = duration*0.034/2; 

if (distance» D min) { 

12| digitalWrite (Buzzer, LOW); 

13| rtdb.setString( *Ultrasonic/SD.status", “Respected” ); } 
14| else {digitalWrite (Buzzer, HIGH); 

15| rtdb.setString(‘‘Ultrasonic/SD_status”, “Not Respected”); } 


= = 
rco -co0o-1c»cmÓ ocrtv€.-w- 


5.4 Results and Discussion 


Our system is operational after interconnecting all hardware and deploying soft- 
ware. Figure 6a represents our real-time database. It is composed of two parts. 
The first part represents the real collected data of COVID-19 patients. Patient 
01 body temperature reached 35.1?, with a heart rate of 75 per minute and 
9896 blood oxygen saturation. The second part contains all the users who have 
access to the database. A user is identified by his email address, full name, and 
phone number. For security reasons, the Firebase IoT Cloud platform offers the 
ability to manage users. As an example, we can delete a user who doesn't have 
the permission to consult the data of a patient. And the most important part 
of our application is that the doctor can visualize the collected data via the 
Android application, as shown in Fig. 6b. Table2 shows the status of a patient 
depending on his measured oxygen saturation. Table3 summarizes the variation 
of heart rate by age. By comparing the collected data with the reference values 
in Table2 and Table3, the doctor can generate an alert such as the need for 
oxygen. Figure 7 illustrates real-time sensor data monitoring for a smart system 
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Fig. 7. IoT monitoring every 


hour (a): Patient 1 (b): Patient 2 (c): Patient 3 


that remotely monitors COVID-19 patients. Curves a, b, and c in Fig. 7 show the 
history of real-time measurements of oxygen saturation and body temperature of 


three COVID-19 patients with 


different ages, every hour from 8:00 a.m. to 21:00 


p.m.. In every plot, the X-axis refers to the time and the Y1-axis, Y2-axis to the 
corresponding temperature and oxygen saturation respectively of the patient. 
We choose three patients to validate our system by comparing our collected 
data with the data of a commercial device, we find an acceptable divergence of 


+ 0.3 °C for temperature and 4 
is incurring a total cost of 13 €, 
is incurring costs of 25 €. Our 


t 1 for SpOs. With this accuracy, our medical kit 
whereas, on the other hand, a commercial device 
application respects the general architecture and 


the three dimensions of the IoT: 


1. Any time connection: we can visualize the data in a real-time way at any 


time. 


2. Any where connection: we can see the data in any place in the world. 


3. Any thing connection: w 


e are able to interconnect all the hardware. 
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Table 2. SpOs reference value Table 3. Heart rate reference value 
Range 94%-98% | Less than 90% Age range (yr)|HR (beat/min) 
Interpretation|Normal | Bad 10-19 80 + 10 

20-29 79 + 10 
30-39 7847 
40-49 T8347 
50-59 76+ 9 
60-69 77+9 
70-79 72+9 
80-99 73 + 10 


The originality of our solution lies in the fact that it resolves some problems 
that we found in previous studies. For example, the authors in [3] propose an 
IoT solution to COVID-19, which is not practical in all situations because they 
developed an application that works only in India. The main disadvantage of 
the system developed by [11], is that authors use a local server instead of the 
cloud, which is not recommended in an IoT system (storage and processing 
constraints). In a recent paper by [4], authors merge the application layer with 
the processing layer by exploiting the dashboard (data visualization) offered by 
the Thing Speak Cloud platform. The first problem is that the system does not 
respect the general architecture of the IoT. The second one is that all users need 
to have access to the cloud platform in order to visualize the collected data. For 
that, users will have full control over data (see all data even if they don’t have 
permission, alter data, etc.), and this is not good for the privacy of patient data. 
Also, we can’t manage users. However, most of the previous studies do not take 
into account the modeling part of the proposed system. 

A part of security is guarantees using the blacklist and whitelist patterns. 
This will lead to establishing a trusted communication partner. Also, we share 
the key and the link of the database with trusted devices. We have complete 
control over the users who have access to patient data to maintain confidentiality. 


6 Conclusion 


In this paper, we presented our proposed meta-model that permits modeling 
the general architecture of the IoT. We implemented two functional applica- 
tions to help the world in this pandemic and outbreak the spreading of COVID- 
19. Firstly, we presented the necessary equipment in both scenarios. Next, we 
designed correct models by construction using UML diagrams to describe the 
behavioral and structural views of our system. After that, we implemented the 
system using the Embedded C (to program the Node MCU) and Java (to pro- 
gram an Android application) languages. Finally, we presented and evaluated 
our application. The application has shown that it is effective in fighting the 
coronavirus with the obtained results. 
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In the short term, we will strengthen our application by adding more sensors 
to offer more services and actuators to have a good preventive and alerting 
system. 

In the long term, we will integrate new technologies such as artificial intelli- 
gence to get the most out of the huge amounts of data that sensors collect. This 
will allow us to make highly precise diagnoses and reactions. 
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Abstract. For medical treatments, pain is often measured by self- 
report. However, the current subjective pain assessment highly depends 
on the patient's response and is therefore unreliable. In this paper, we 
propose a physiological-signals-based objective pain recognition method 
that can extract new features, which have never been discovered in 
pain detection, from electrodermal activity (EDA) and electrocardio- 
gram (ECG) signals. To discriminate the absence and presence of pain, 
we establish four classification tasks and build four tree-based classifiers, 
including Random Forest, Adaptive Boosting (AdaBoost), eXtreme Gra- 
dient Boosting (XGBoost), and TabNet. The comparative experiments 
demonstrate that our method using the EDA and ECG features yields 
accurate classification results. Furthermore, the TabNet achieves a large 
accuracy improvement using our ECG features and a classification accu- 
racy of 94.5196 using the features selected from the fusion of the two 
signals. 


Keywords: Pain detection - Physiological signals * Classifier - TabNet 


1 Introduction 


Pain is a precious tool for medical attention. A widely accepted definition con- 
siders pain as an unpleasant sensory and emotional experience associated with 
actual or potential tissue damage, or described in terms of such damage [1]. Pain 
helps identify harmful situations, avoid tissue damage, and promote healing [2]. 
While each individual learns pain through previous experiences related to injury, 
which can confound subjective pain ratings [3]. The criterion for evaluating pain 
can be different for each person. Consequently, accurate assessment of pain is 
challenging due to the subjectivity of pain. 

As reported in [4], pain is a complex and not entirely understood phe- 
nomenon. Self-report is used as the golden measurement, which is however not 
always reliable and valid [5]. For instance, assessing pain from children depends 
on the cognitive development, clinical context, and pain typology [6]. There- 
fore, the awareness of pain may vary among different ages of children. For the 


© The Author(s) 2022 
H. Aloulou et al. (Eds.): ICOST 2022, LNCS 13287, pp. 183-195, 2022. 
https://doi.org/10.1007/978-3-031-09593-1 14 


184 H. Shi et al. 


elderly, effective pain evaluation can be a challenge [7]. Moreover, pain may not 
be self-reported in some situations. For example, neonates cannot utter words 
to describe their pain [8]. Individuals who have no clear consciousness of their 
feelings, like dementia [9], cannot report their pain neither. An objective system 
for automatically recognizing pain can contribute to detecting pain in such situa- 
tions. This in turn helps improve people’s physical health and mental happiness. 

A bunch of machine learning methods have been developed for pain recog- 
nition, such as random forest [11] and support vector machines (SVM) [10]. 
However, these classifiers rely on the selected features and empirical knowledge 
and therefore suffer from the complexity of the tasks. Nowadays, it is well known 
that deep learning can provide accurate recognition or prediction for complex 
data science problems [12] as it allows discovering hidden patterns in data and 
modelling complex relationships between variables. 

In light of the above statement, we explore four tree-based classification mod- 
els, i.e., eXtreme Gradient Boosting (XGBoost) [13], random forest, Adaptive 
Boosting (AdaBoost) [14], and TabNet (which is the recently proposed deep- 
learning-based ensemble decision trees) [15]. To improve the classification accu- 
racy, we extract new electrodermal activity (EDA) features and electrocardio- 
gram (ECG) features of physiological signals. The main contributions of this 
work include: 1) the extracted novel features from biomedical signals, which can 
boost the current models in pain assessment; 2) the application of TabNet model, 
which is the first attempt in pain detection; 3) the outstanding performance of 
TabNet for most detection tasks, compared to the existing models. 


2 Related Work 


In this section, we will introduce the previous studies that are related to this 
work. All the related work mentioned below with better performance will be 
compared to our method in Sect. 4.2. 

Early research in pain assessment was mainly focused on fusion of multi- 
model signals, involving facial expression and biomedical signals based on BioVid 
Heat Pain Database (BVDB) (Part-A) [16]. For example, Werner et al. [17] 
applied a random forest classifier on multi-model signals to detect pain level. 
Káchele et al. [18,19] conducted random forest classifier for continuous predic- 
tion of pain intensity. However, facial expression-based pain recognition needs 
tracking of the facial regions, which can be cumbersome and complex in clin- 
ical application. Pain research indicates that the presence of pain significantly 
interacts with autonomic nervous system and thus leads to changes in EDA and 
heart rate [20]. The bio-physiological signals are recorded in BVDB as follows: 
1) EDA, which is also referred to as skin conductance (SC) or galvanic skin 
response (GSR), measures the changes in the electrical properties of the skin 
and shows the strong association to the emotional arousal [16]; 2) ECG records 
the electrical activity of the heart and supplies a significant amount of informa- 
tion about heart function [21]; 3) Electromyogram (EMG) records the electrical 
activity of muscles. The activity of the trapezius is a hint of a high stress level 
which is to be expected during pain stimulation [16]. 
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Given the correlation between physiological signals and unpleasant pain, 
nowadays, pain assessment researchers have shifted to focusing on merely phys- 
iological signals. Gruss et al. [22] extracted a total of 159 features and proposed 
SVM classifiers for binary pain classification. Deep learning models on physiolog- 
ical signals have shown promising results in pain recognition. In Lopez-Martinez 
et al. [23], EDA and ECG signals features were introduced into multi-task neu- 
ral networks which have two hidden layers, one shared and one person-specific, 
and proved to perform better than single-task neural networks. Wang et al. [24] 
proposed deep Recurrent Neural Network based hybrid classifiers to classify the 
pain intensity. They applied a bidirectional Long Short-Term Memory (LSTM) 
network to learn temporal dynamic characteristics of physiological signals and 
fused them with handcrafted features. Thiam et al. [25] proposed a multi-modal 
information aggregation approach based on Deep Denoising Convolutional Auto- 
Encoders for pain assessment on two different pain databases including BioVid 
Heat Pain Database. Thaim et al. [26] designed Convolutional Neural Network 
(CNN) based on physiological signals (EDA, ECG, and EMG) for pain classi- 
fication. Pouromran et al. [27] computed features from EDA, ECG and EMG 
signals and trained machine learning models—including Linear Regression, Sup- 
port Vector Regression, Neural Network and XGBoost—on these features for 
pain intensity estimation. Subramaniam et al. [28] proposed a hybrid CNN- 
LSTM classification based on ECG and EDA signals for binary pain detection. 

In [26], the pain classification with EDA signals was much higher than other 
signals and the fusion of EDA and ECG signals. However in this work with our 
new model, we observed different results. For ECG signals, our extracted features 
provide additional information allowing to improve pain detection. For instance, 
by comparing the results of the classification task Bo vs P4 using random forest 
classifier, we notice that Kachele et al. [18,19] provided an accuracy of 53.90%, 
Werner et al. [17] 6296, while we were able to generate an average accuracy of 
67.1896 using our new ECG features. Furthermore, using the TabNet model with 
ECG features, we were able to improve the accuracy to 81.12%. 


3 Proposed Model 


3.1 Biovid Heat Pain Database (BVDB) 


The BioVid Heat Pain Database [16] contains multidimensional datasets, both 
video signals and biopotentials, which provide potential to advance an auto- 
mated pain recognition system. The data was collected on 90 subjects from 18 
to 65 years of age, using a thermode at right arm for the pain elicitation. The 
experimenters randomly heated the participants with four calibrated intensities 
(T1, T2, T3, T4) and each stimulation was held up to 4s (Fig. 1a). Thus, it gener- 
ates four pain intensity levels (i.e., P,, P5, P3, P4). 20 times of each pain level were 
given. The pauses between the stimuli were kept at baseline temperature (To) 
and around 8 s-12 s, which was no pain (Bo). Each dataset was extracted with 
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Fig. 1. (a) Pain stimulation, represented by the heat levels from To to T4. The features 
are extracted from the blue window of length 5.5s. (b) The standard ECG signal with 
P, Q, R points and amplitudes. 


the length of 5.5s starting 1s after the stimulation was given, where 2816 data 
points were collected. Consequently, the experimenters retained 20 samples/class 
x 5 classes — 100 samples for per subject. 


3.2 Feature Extraction 


BVDB includes five parts, Part A-E [29]. Besides Part-A, we also explored our 
methods on Part-B with EDA and ECG signals. Before extracting features from 
ECG signals, a Butterworth bandpass filter with frequency range of [0.1, 250] 
was introduced to remove the noise of the muscles, baseline wander and other 
interference [30]. In Part-B, 6 samples contain a quantity of nulls with EDA 
signals. Their corresponding ECG signals are also invalid. Then we removed the 
6 samples in our experiment, which are all from no pain stimulation (Bo). 

In addition to utilizing EDA and ECG features proposed in [23], we explored 
novel features. We found that some EDA signals are like the sine function and 
some are not, so we investigated the numbers of crests and troughs, and the 
relations between the start and end states, minimum and maximum. These four 
SC features were appended. On the other hand, ECG signals are mainly reflected 
on the characteristics of P, R and T points. ECG signals consist of the P-QRS- 
T waves and P, R, T points are the peaks of P, R, T waves (Fig. 1b) [31]. We 
detected false R points with visualization and removed them via their amplitudes 
because these points may seriously impact on the preciseness of further feature 
extraction. We extract the information of R, P, and T points as ECG features. 


Skin Conductance (SC). We applied 12 SC features in [23]. (1) maxi- 
mum; (2) range; (3) standard deviation; (4) interquartile range; (5) root mean 
square; (6) mean; (7) mean absolute value of the first differences (Mavld): 
LX a |vj41 — zil; (8) means absolute value of the second differences: 


Tree-Based Models for Pain Detection from Biomedical Signals 187 


ws eee |zi+2 — zil; (9) mean absolute value of the first differences of the 


standardized signal: E (10) mean absolute value of the second differ- 


ences of the standardized signal; (11) skewness; (12) kurtosis were computed. 
In addition, we appended the novel features: (13) differences among maximum, 
first state and last state (Maxln): maximum — zı — zw; (14) the number of 
troughs; (15) differences between minimum, first state and last state (N1Min): 
XN — zı — minimum and (16) the number of crests. The abbreviation of SC 
features are summarized in Table 1. 


Electrocardiogram. First, we located P, R, T points. Their amplitudes and 
positions were kept. The R points were detected by Pan-Tompkins algorithm 
[32]. After locating the R peaks, the area of T wave was determined from R() + 
0.16 x RR(i -- 1) to R(i) +0.57 x RR(i+1) [21], where R(i) is the i^^ position of 
R point; RR represents RR interval, RR(i+ 1) is achieved from R(i 4- 1) — R(i). 
'The peak of phase signal within the segment was considered as the T' points. 
Finally, in the similar way, P waves were demarcated from R(i)+0.7 x RR(i-4- 1) 
to R(1) -- 0.07 x RR(i-- 1) [21], where the range was modified to make it suitable 
for datasets from BVDB. The P, R, T points were correctly located and a sample 
of ECG signals is shown in Fig. 2. 

Second, we calculated the following ECG features. The 4 of 5 ECG features 
proposed in [23] are included in our work. They are (17) the mean of the IBIs; 
(18) the root mean square of the successive differences (RMSSD); (19) the mean 
of the standard deviations of the IBIs (SDNN); (20) the mean of the slope of the 
liner regression of IBIs in its time series; (21) the ratio of SDNN to RMSSD. New 
features that we computed are (22) the number of R peaks in the windows of 
5.5s (Fig. la); (23) the range of R amplitudes; (24) the standard deviation of R 
amplitudes; (25) the mean of the duration of PT (Fig 1b): -4 SETTA- P), 
where M is the minimum of the number of P points and T points, T'(i) is the it” 


Raw ecg data with pulse stream 


— T-points 
—— P-points 
15004 — Q-points 
—— R-peaks 
|—— S-points 


Amplitude (mV) 


Time 


Fig. 2. Raw ECG signal of a sample with detected T, P, Q, R, S points. 
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Table 1. Extracted features 


Signal | Abbreviation of extracted features 


EDA | 1. Max, 2. Range, 3. STD, 4. IQR, 5. RMS, 6. Mean, 7. Mavld, 8. Mav2d, 
9. Mavidss, 10. Mav2dss, 11. Skew, 12. Kurt, 13. Max1n, 14. #Trough, 
15. N1Min, 16. #Crest 

ECG |17. MIBIs, 18. RMSSD, 19. SDNN, 20. MSlope, 21. Ratio, 22. #Rpeaks, 
23. RangeRy, 24. STDRy, 25. PTxM, 26. PTxRMSSD, 27. PTxSDNN, 
28. PTxRatio, 29. PTyM, 30. PTyRMSSD, 31. PTySDNN, 32. PTyRatio 


position of T point, P(i) is the it” position of P point; (26) the root mean square 
of the successive differences of the duration of PT (PTxRMSSD); (27) the mean 
of the standard deviation of the duration of PT (PTxSDNN); (28) the ration 
of PTxSDNN to PTxRMSSD; (29) the mean of the difference of amplitude of 
PT: à Yi (Amplitude(T;) — Amplitude(P;)); (30) the root mean square of 
the successive differences of the amplitude of PT (PTyRMSSD); (31) the mean 
of the standard deviation of the amplitude of PT (PTySDNN); (32) the ration 
of PTySDNN to PTyRMSSD. They are summarized in Table 1. 


3.8 Methodology 


The main goal is to discriminate between no pain (Bg) and the presence of pain 
(Pı, P», P3, P4) based on physiological signals. For this, we establish four binary 
classification tasks. In the tasks, we extract features from EDA and ECG sig- 
nals — i.e., create tabular EDA and ECG features. Based on these features, we 
explore the decision trees (DTs) based classifiers, i.e., random forest, AdaBoost, 
XGBoost, and TabNet. In general, ensemble methods tend to yield better results 
than standard D'Ts. For this reason, we choose tree-based tabular data learn- 
ing methods. Random forest [11] is an ensemble of decorrelated decision trees. 
Adaboost is a classic type of Boosting [14]. Boosting is a general ensemble 
method that produces a strong classifier from an ensemble of weak learners. 
XGBoost [13] is an ensemble DT approaches that follows the principle of boost- 
ing. TabNet [15] is the integration of deep neural network into DTs. The archi- 
tecture of TabNet contains a Batch Normalisation layer to filter the raw data, 
and several transformer blocks to learn relevant features. It also consists of a 
sequential attention mechanism and learnable masks to choose which feature to 
process at each decision step. This characteristics enables efficient learning as the 
learning capacity is used for the most salient features. Finally, we can select the 
best combination of signals, features, and models based on their performance. 


4 Experiments 


4.1 Setup 


On the Part-A and Part-B datasets, we conducted four binary classification 
tasks of discriminating subjects with no pain (Bo) versus pain (Pi, P», P3, P4) 
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condition, i.e., Bo vs. Pj, Bo vs. Po, Bo vs. P3, and Bo vs. Py. Since Part-A 
includes 87 subjects and Part-B 86 subjects, we have 87 x 20 x 2 = 3480 samples 
for each of the four Part-A tasks and (86 x 20 — 6) + 86 x 20 = 3434 samples 
for each Part-B task. We applied the PyTorch implementation of TabNet [15], 
which is available at [33]. We evaluated the model’s performance via the stratified 
10-fold cross-validation because the Part-B data is imbalanced due to 6 invalid 
samples in no pain level (Bo). To improve the models’ performance, we employ 
the grid search optimization method to select the best hyperparameters from 
the following hyperparameter candidates: 


Random forest: The n_estimators is in range 60 to 280 with a 20 step size. 
e AdaBoost: Given the decision tree base. estimator, the optimizer determines 
the max depth € {10,11}. The n estimators takes a value in the range [90, 
240] and the learning .rate from (0.001, 0.01, 0.5, 0.1, 1}. 

XGBoost: With max depth—7, the value of n_estimators is selected from 
(100, 200, 300, 400, 500}, learning rate from (0.01, 0.1, 0.2, 0.3, 0.7}, and 
gamma from (0.1, 0.2, 1, 2, 5]. 

e TabNet: For TabNetClassifier [15], the batch. size is either 8% or 10% of the 
total training dataset size. For learning rate schedulers, we set gamma=0.9, 
step_size=10. A validation-based early stopping strategy was employed. 


4.2 Results and Discussion 


We present and discuss classification results with respect to Part-A and Part- 
B in this section. Since it's the first attempt to investigate machine learning 
algorithms on Part-B, we only compare our work with previous works involving 
physiological signals of Part-A. 


e The pain detection performance of the four models on Part-A is summarized 
in Table2. It can be seen that the best accuracy for the four tasks is 65.57% 
for Bo vs. P, task, 68.39% for Bo vs. P2, 76.15% for Bo vs. P3, and 85.23% for 
Bo vs. P4, respectively, when EDA is used. TabNet steadily outperforms the 
random forest and Adaboost classifiers. For lowest pain level (Pj), TabNet 
is superior to XGBoost. For higher pain levels (P5, P3, and P4), XGBoost 
performs better than TabNet. For ECG, TabNet consistently outperforms all 
the other models, achieving the 72.18%, 71.81%, 77.04%, and 81.12% accuracy 
for the four tasks, respectively. TabNet wins the competition when it uses the 
fusion of EDA and ECG signals, with the accuracy of 75.71%, 83.97%, 88.93%, 
and 94.51% for the four tasks, respectively. 

e The four models for four classification tasks (Bo vs. P, Bo vs. P3, Bo vs. P3, 
Bo vs. P4), the results on Part-B are presented in Table3. For EDA signals, 
the best accuracy is 61.18%, 64.83%, 69.22%, and 79.22%, respectively. Tab- 
Net performs better than other classifiers on the lowest pain level (Pi). For 
higher three pain levels (P2, P3, P4), XGBoost generates the best accuray. In 
addition, TabNet did the best for the eights tasks (ECG and EDA+ECG). 
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Table 2. Pain recognition with physiological signals of Part-A. Mean accuracies are 
reported for stratified 10-fold cross-validation, for representative models and binary 
classification tasks (Mean% + Standard Deviation%). 


Pain intensity | Classifier | EDA ECG EDA + ECG 

Bo vs P4 RF 82.09 + 1.78 | 67.18+0.98 | 83.514 1.92 
AdaBoost | 83.82 + 0.68 |68.51 +2.02 | 86.67 + 0.80 
XGBoost | 85.23 + 0.80 | 69.37 + 1.54 88.07 + 1.29 
TabNet 83.99+ 2.13 |81.12 + 1.86 | 94.51 + 0.43 
Bo vs P3 RF 71.9242.82 61.35 +2.86 | 73.70 + 2.69 
AdaBoost | 75.23 + 1.83 |62.93+1.31 |78.45+0 
XGBoost | 76.15 + 1.36 | 64.86 + 0.13 | 78.16 + 2.06 
TabNet 74.54 +2.96 |77.04 + 1.86 88.934 1.33 
Bo vs P> RF 63.68 + 2.08 | 55.86 +1.73 65.34 + 2.04 
AdaBoost | 66.41 + 0.78 | 58.79 +1.35 | 67.76 + 0.65 
XGBoost | 68.39 +0 58.51 +0.73 169.540 
'TabNet 67.76 + 2.2 71.81 + 1.71 83.97 + 1.78 
Bo vs Pi RF 55.69 + 2.31 |51.23 +1.56 56.72 + 1.27 
AdaBoost | 60.34 + 0 57.01 +0.26 | 60.26 + 0.98 
XGBoost | 61.49 +0.86 |57.98 +0.86 |61.49+0 
TabNet 65.57 + 2.52 | 72.18 + 1.99 | 75.71 + 2.02 


The best accuray is 69.19%, 74.34%, 76.82%, and 82.85% with ECG signals, 
and 75.24%, 78.74%, 82.44% and 88.44% with the fusion of EDA and ECG. 
It shows the same trend as Part-A. 


In Table 4, we compare our results with previous studies on single signals of 
Part-A for classifying no pain and highest pain levels. For EDA signal, Thiam 
et al. [26] applied CNN algorithm with an average accuracy of 84.57% and 
achieved the highest accuracy among previous work. For XGBoost, we achieved 
an accuracy of 85.23%. TabNet outperforms the most previous work as well. For 
ECG signal, TabNet with our selected features yields 72.18%, 71.81%, 77.04% 
and 81.12% accuracy for the four tasks, outperforming the currently best model 
[28] (with an accuracy of 68.7%, 62.61%, 67.86%, and 75.21%). The performance 
with ECG signals has been significantly improved. From these results, we found 
that ECG signals are essential for pain detection, which are comparable to EDA 
signals; this finding has never been previously reported. 

Table 5 shows the comparison between the best binary classification accuracy 
reported by the previous studies and ours, for discriminating no pain (Bo) and 
pain tolerance (P4) on Part-A. The facial expression (Video) turns out to be 
non-promotional for pain detection, according to results shown in [17-19]. On 
the contrary, using physiological signals, especially EDA and ECG signals, can 
boost the pain detection. Additionally, using the fusion of EDA and ECG signal, 
TabNet achieves the best pain detection results, with an accuracy of 94.51%. 
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Table 3. Pain recognition with physiological signals of Part-B. Mean accuracies are 
reported for stratified 10-fold cross-validation, for representative models and binary 
classification tasks (Mean% + Standard Deviation%). 


Pain intensity Classifier | SCL ECG SCL + ECG 
Bo vs P4 RF 75.45 +2.23 |64.38+1.62 | 76.97 + 2.60 
AdaBoost | 77.05 + 1.12 |68.04+1.09 | 79.01 4 0.12 
XGBoost | 79.22 + 1.12 | 68.36 + 1.24 | 81.04+ 1.86 
TabNet 77.11 +2.46 | 82.85+ 2.64 | 88.44 + 1.52 
Bo vs P3 RF 66.39 + 2.42 159.96 +1.75 68.314 1.41 
AdaBoost|67.72-- 1.10 63.43 + 1.62 | 72.27 + 0.38 
XGBoost | 69.22 + 0.49 | 62.28 + 0.62 | 70.61 + 2.12 
'TabNet 68.70 +2.80 | 76.82 + 1.59 | 82.44 + 3.02 
Bo vs P> RF 59.58 + 2.18 (53.78 +1.80 59.93 + 2.55 
AdaBoost | 64.53 + 0.0 56.84 1.10 | 62.31 + 0.39 
XGBoost | 64.83 + 1.19 | 58.72 + 1.74 | 64.13 + 0.40 
TabNet 63.02 + 2.24 | 74.34 + 2.68 78.74 + 2.54 
Bo vs Pi RF 55.64 + 1.62 |52.10 +3.49 | 54.60 + 1.81 
AdaBoost | 59.62 + 0.46 |59.06 +0.46 | 55.29 + 1.88 
XGBoost | 60.81 +2.21 |56.08 +0.28 60.02 + 1.65 
TabNet 61.18 + 1.35 | 69.19 + 2.38 75.24 + 2.34 


Table 4. Performance comparison of the binary classification task Bo vs. P4 


Study EDA ECG 
Wernel et al. [17] 73.80 62.00 
Lopez-Martinez et al. [23] | 79.98 62.50 
Kächele et al. [18,19] 81.10 53.90 
Thiam et al. [26] 84.57 57.04 
Subramaniam et al. [28] | 80.17 75.21 
Our Method TabNet: 83.99 |81.12 
XGBoost: 85.23 | 69.37 


In order to investigate the contribution of EDA and ECG features for the 
classification of no pain and pain tolerance, we plot the significance of the fea- 
tures extracted from the fusion of EDA and ECG in using XGBoost (Fig. 3a) 
and TabNet (Fig. 3b). Based on the XGBoost’s feature importances, we find that 
the extracted EDA features contribute 65% to the classification, while ECG con- 
tributes 35%. (The computation is not explicitly presented here due to the space 
limit.) Moreover, the combination of the 4 features from EDA (i.e., the features 
#13-16 in Table 1) and the 10 from ECG (i.e., features #23-32 in Table 1) con- 
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Table 5. Comparing the best binary classification accuracy of each study on Part-A 


Study Signal (or Sensor) Bo vs. P4 
Wernel et al. [17] EDA+ECG+ EMG-+ Video | 80.6 
Lopez-Martinez et al. [23] | EDA+ECG 82.75 
Kachele et al. [18,19] EDA+ECG+ EMG+ Video | 83.1 
Wang et al. [24] EDA+ECG+ EMG 83.3 
Pouromran et al. [27] EDA 83.3 
Thiam et al. [25] EDA+ECG+ EMG 84.25 
Subramaniam et al. [28] | EDA+ECG 94.12 
Our Method+TabNet | EDA+ECG 94.51 


tribute 37.48% to the classification. For TabNet, the extracted EDA and ECG 
features make a contribution of 61.15% and 38.85%, respectively, while our 4 
new EDA features and 10 ECG features contribute 39.76% to the classification. 
The comparison reveals that 1) besides EDA, ECG is significant for the pain 
detection; 2) important features have been overlooked by previous researches. 


Feature Importances 


Feature Importances 
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(a) XGBoost using our selected features 
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(b) TabNet using our selected features 


Fig. 3. The importance of features (EDA and ECG) determined by XGBoost(a) and 
TabNet(b) for the classification of Bo and P4. 


5 Conclusion 


We proposed in this paper an automatic pain detection method for the clas- 
sification of subjects with no pain and pain condition. For each of the four 
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classification tasks, we explored the performance of four tree-based models that 
are learned based on the features extracted from a single physiological signal 
(EDA and ECG) and from their fusion (EDA + ECG). We removed the noise of 
ECG signals and appended corresponding P-QRS-T wave information to ECG 
features. Our method can close the gap to previous work and discover signifi- 
cance of ECG in pain detection. The experimental results demonstrate that our 
method achieves the highest classification accuracy when using a single signal 
(EDA or ECG). The 94.51% accuracy on the data of physiological signals reveals 
the promising of our method in practical use — e.g., detecting pain based on the 
real-time data collected from wearable devices. 
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Abstract. To improve the QoL of the elderly, it is essential to pre- 
dict their stress states. In general, the stress state varies from day to 
day or time to time depending on what activities are performed and 
how long/strong. However, most existing studies predict the stress state 
using biometric data and specific activities (e.g., sleep time, exercise time 
and amount) as explanatory variables, but do not consider all daily liv- 
ing activities. Therefore, it is necessary to predict the stress state by 
linking various daily living activities and biometric information. In this 
paper, we propose a method to improve the prediction accuracy of stress 
estimation by linking daily living activities data and biometric data. 
Specifically, we construct a machine learning model in which the objec- 
tive variable is the result of a stress status questionnaire obtained every 
morning and evening, and the explanatory variables are the types of 
daily living activities performed in the 24h prior to the questionnaire 
and the feature values calculated from the biometric data during each of 
the performed activities. The results of the evaluation experiments using 
the one month data collected from five elderly households, show that the 
proposed method (using per-activity biometric features) improves the 
prediction accuracy by more than 10% from the baseline methods (with 
biometric features without considering activities). 


Keywords: Health care - Smart home - Wearable sensors - Stress 
prediction 


1 Introduction 


With the aging of the population in developed countries, there is an urgent need 
for effective measures to promote the care of the elderly and the extension of 
their healthy life expectancy. However, there are many elderly people who wish 
to continue to live independently at home. To grant this wish, it is important 
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to build an environment in which elderly people can understand their daily 
activities and manage their health conditions by themselves. There have been 
many studies on activities sensing/recognition technologies in the home and 
health condition monitoring using these technologies to build an environment 
for self health management and improve lifestyle habits of the elderly [1]. 

In order to provide monitoring services and improve the lifestyle of the elderly, 
it is necessary to develop indicators of health status and investigate the factors 
that cause these indicators to change. Currently, the World Health Organization 
(WHO) has developed a questionnaire index called WHOQOL-100 [2] as a repre- 
sentative indicator of health status. However, the method of measurement using 
such a questionnaire index places a heavy burden on respondents due to a large 
number of questions. For this reason Amenomori et al. [3] conducted a study 
to measure HRQOL (Health Related Quality of Life) with less burden by using 
devices such as smartphones and smartwatches. They aimed to improve HRQOL 
which is strongly related to physical and mental stress states, by continuously 
measuring HRQOL and detecting signs of stress early to prevent it. 

Stress estimation has also been studied using devices rather than question- 
naires to reduce the burden of estimation. Stress estimation using devices has 
been applied to many situations such as daily life [4-6]. Fukuda et al. [4] col- 
lected wake-up times and sleep data obtained from wearable devices and answers 
of occupational health and safety questionnaire called the Depression and Anx- 
iety Mood Scale (DAMS) in office workers for two to three weeks at a general 
company, and constructed a machine learning model to estimate the question- 
naire answer. In addition, Natasha et al. [5] proposed a method to quantitatively 
measure and predict the health, stress, and happiness of the next day using smart 
devices, respectively, in order to understand the stress state at work. 

In this paper, as a method to improve the prediction accuracy of stress esti- 
mation, we propose a method to predict stress by linking daily living activities 
data and biometric data. Specifically, we construct a machine learning model 
with the results of a stress status questionnaire obtained every morning and 
evening as the objective variable, and the types of daily activities performed 
in the 24h before the questionnaire and the features calculated from biometric 
data during each daily activity as the explanatory variables. As biometric data, 
we use the Lorenz plot area calculated from the heart rate data collected by a 
smartwatch (Fitbit). The Lorenz plot area is known to be useful for stress esti- 
mation because it can visualize the activity level of the parasympathetic nervous 
system [7]. 

'To evaluate the effectiveness of the proposed method, we applied the method 
to the dataset [1] consisting of daily living data, biometric data, and stress 
status questionnaires (two questions in the morning and two questions in the 
evening) collected from five households of elderly people over 60 years old for 
one month. To confirm the validity of the per-activity biometric features, we 
compared the baseline method 1 (using 24-h RRI variance and Lorenz plot area 
[7,8] as features), the baseline method 2 (adding sleep time, which has been 
validated in the previous study [4], as a feature to the baseline method 1), 
and the proposed method (using RRI variance and Lorenz plot area for each 
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activity type as features). For three class classification, the accuracy of the four 
questionnaires on average was 34.5% for the baseline method 1 and 47.25% for 
the baseline method 2, while the average accuracy of the proposed method was 
58.25%, confirming the effectiveness of considering the per-activity biological 
features. 


2 Related Research 


This section surveys existing studies on QoL estimation, stress estimation and 
health management using smart home technologies. 


2.1 QOL Estimation 


Quality of life (QoL) is a measure of satisfaction and quality in the daily life. The 
QoL [9] research originally started as a concept to discuss the quality of life after 
treatment in the medical field, but it is now used not only in the medical field 
but also as a concept related to the quality of life in general, such as work-life 
balance and happiness. 

In particular, QoL, which is directly related to human health, is called 
HRQOL (Health Related Quality of Life), and is evaluated by categorizing it into 
various domains such as physical, psychological, social interaction, economic and 
occupational, and religious and spiritual states. The World Health Organization 
(WHO) has developed various indicators to quantitatively assess HRQOL, such 
as WHOQOL [2] and Short Form [10]. These indicators are assessed using paper 
questionnaires. However, the WHOQOL-100 [2] requires 100 items in 6 domains, 
while the SF-36 [10] requires 36 items in 8 domains. The labor to answer these 
questions make it difficult to assess the quality of life on a daily basis. 

Amenomori [3] et al. have proposed a method to continuously measure 
HRQOL using mobility and biometric information obtained from smartphones 
and smartwatches, and have shown that HRQOL can be predicted using a small 
number of questionnaires and information from smart devices. They aimed to 
improve HRQOL by continuously measuring HRQOL, which is strongly related 
to physical and mental stress states, to detect early signs of stress and to prevent 
it. 

Unlike Amenomori's work, the goal of our work in this paper is to detect 
and prevent causes of daily living stress in the elderly at an early stage. For this 
goal, we try to use per-activity biometric information to predict the stress state 
so as to understand which activity causes the stress. 


2.2 Stress Estimation 


There have been many studies on stress estimation using devices in many situ- 
ations. There are two groups of stress estimation. The first group is the studies 
conducted in a controlled laboratory environment [11]. In these studies, the 
researchers intentionally generated stress using some kind of stress test, where 
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the researcher has complete control over the stress level, and high-stress detec- 
tion accuracy (80-97%) is usually reported. 

The second group is studies that analyze stress in real life [4,5,12,13] which 
have reported low accuracy [14]. Therefore, various systems are being researched 
to improve the prediction accuracy. To analyze stress in workers, not only smart- 
watches but also chest-mounted heart rate sensors (e.g., WHS-3 [15]) have been 
used [16]. However, systems that use multiple wearable sensors place a high bur- 
den on the subject and are not suitable for real-time stress detection in daily 
living. Therefore, it is desirable to use wearable devices as minimum and unob- 
trusive as possible. 


2.8 Health Management in Smart Home 


Since the elderly spend more time at home than out of the house, there has 
been development of systems to manage health conditions using activities data 
in the home, such as smart homes. Some smart home research has attempted to 
correlate activities and predict the well-being of occupants within a living space. 
The goal of Intel Research's Computer-Supported Coordinated Care (CSCC) 
project [17] is to identify the care network characteristics and needs of older 
adults who wish to live at home. In addition, Jakkula et al. [6] aimed to identify 
trends in health status over time, with the goal of creating a smart home system 
focused on next-generation health care capable of home health care. However, 
most existing studies related to stress estimation in the smart home have been 
conducted in a controlled smart home, not in the subject's home or other ordi- 
nary residence. Therefore, there are few studies that analyze stress in the home 
in real life. 

In addition, in most of the related studies described above, the sensor data 
and feature values used do not take into account all aspects of the subject’s home 
life. Therefore, if there is no information related to activities of daily living, it 
is not possible to understand the activities that causes stress, and it may not 
be possible to improve the stress state. Therefore, it is necessary to incorporate 
data related to activities and biometric information in the home. 


3 Proposed Method 


In this study, we propose a method for estimating the stress state at the end of 
a day (or the beginning of the next day) from the values of the stress index for 
each of daily living activities which are performed in a day. In the following sub- 
sections, we describe the method of obtaining daily living activities and stress 
indicators used in the proposed method, the definition of the stress state to be 
predicted, and the method of constructing the model. 


3.1 Acquisition of Daily Living Activities 


In this study, we assume an environment in which the activities of residents can 
be automatically obtained by using an in-home daily living activities recognition 
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Fig. 1. Components of SALON system. 


system (such as SALON [18]). We target primary activities such as cook, meal, 
rest, work, clean, wash, go out, bath, and sleep. We assume that the in-home 
activity estimation system records the time of day when each activity is per- 
formed. Table 1 shows an example of activities log. The SALON system does the 
following to stabilize the annotation accuracy. The SALON system places anno- 
tation buttons near the activity location with an aim to improving annotation 
accuracy. When a user forgets to press the annotation button, annotation data 
is corrected by the user. 


3.2 Acquisition of Stress Index 


In this study, we assume that residents wear a wearable biometric sensor (ECG) 
that can measure heart rate and heart rate variability. In the proposed method, 
we focus on the RRI variance and Lorenz plot area as stress indicators that can 
be calculated from heart rate variability data [7,8]. 

In a Lorenz plot (LP), denoting RR interval at time t by RRI(t), a dot is 
plotted at the coordinate (RRI(t), RRI(t 4- 1)) in a two dimensional plane. The 
area of the ellipse that covers the plotted dots is calculated as the Lorenz plot 
area as shown in Fig. 3. 

On the y = x axis, let m denote the mean of the distance from the origin 
(0,0) (on the y = zx axis), 6, the standard deviation from the origin. On the 
y = —« axis, let 0(.., denote the standard deviation from the origin (0,0). In 
this case, the area of the ellipse with major axis 6, and minor axis Ó( ,j is 
calculated as follows: 

S = T - Öz -6(—z) (1) 


3.3 Association of Activities with Stress Indicators 


In the proposed method, for each activity obtained in Sect. 3.1, the RRI variance 
and Lorenzplot area are calculated from the heart rate data while the activity is 
performed by the method described in Sect. 3.2. If the same activity is performed 
repeatedly in different time intervals (e.g., Cook, Meal and Rest in Table 1), the 
biometric data measured during those intervals are integrated and stress indica- 
tors are calculated for the integrated data. Finally, per-activity stress indicators 
values are calculated as shown in Table 2. 
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Fig. 2. Tabulation procedure for morning and evening questionnaires. 


3.4 Prediction Model Construction 


In this study, we construct a machine learning model to predict the stress state 
where we used as groundtruth the answers for the questions that residents give 
at the end of the day, from per-activity stress indicators values explained in 
Sect. 3.3. 


4 Evaluation Experiment 


For the evaluation experiment, we collected a dataset consisting of activities 
data, biometric data and questionnaire answers. In each home of the elderly 
participants, we set up sensors (up to 10 motion and ambient sensors and a few 
door sensors), five annotation buttons (corresponding to activities) and a server 
of a SALON shown in Fig. 1. In the experiment, first we extract features from 
the dataset, and then construct a random forest classifier to predict the stress 
level. 


4.1 Dataset 


We collected the dataset from five general elderly households (one single, four 
married couples, all in 60's), which consists of daily living activities data, bio- 
metric data, sensor data, and stress state data (obtained through questionnaire). 
The data collection period is one month for each household. The details of each 
data are described below. 
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Fig. 3. How to calculate the Lorenz plot area. 


Daily Living Activities Data. We collected data on five typical daily living 
activities (bathing, cooking, eating, going out, sleeping, and other). Residents 
record the start and end of each activity by pressing the annotation buttons 
Shown in Fig. 1 installed at the locations where the activities are performed. 


Biometric Data. We use the heart rate data collected by a smartwatch (Fitbit 
Alta HR) as biometric data. Fitbit collects the heart rate [bpm] once every 15s 
for one minute. 

We convert the heart rate to RRI [ms] and generate features for stress esti- 
mation, i.e., the variance of RRI and Lorenz plot area. The conversion equation 
for RRI is shown in Eq. (2). 


60 
= (2) 
Heartrate x 1000 


where RRI is defined as the heartbeat interval in ms, and Heartrate is the 
number of heartbeats per minute. 


RRI 


Stress State Data. Stress state data is collected by asking residents to fill out 
questionnaires immediately after waking up and before going to bed each day. We 
adopted four questionnaire items related to mental stress and physical stress. In 
the questionnaire, a five-point Likert scale was used. The specific questionnaire 
items are shown below. 


Questionnaire Items Immediately After Waking Up 


M1. Did you feel physically refreshed this morning? 
M2. Did you feel mentally refreshed this morning? 
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Table 1. Activities log Table 2. Per-activity stress indicators 
Time Activity Activity |RRI variance|LP area 
—7:00 Sleep Sleep 0.8 1.0 
7:30-8:00 Cook Cook 0.3 0.5 
8:00-8:30 Meal Meal 0.2 0.4 
8:30-9:00 Rest Rest 0.3 0.3 
9:00-18:00 | Go Out Go out |0.1 0.3 
18:00-18:30 Cook Bath 0.5 0.7 
18:30-19:30 Meal Sleep 0.9 0.9 
19:30-21:00 | Rest 
21:00-21:30 Bath 
21:30-23:00 | Rest 
23:00- Sleep 


Questionnaire Items Before Bedtime 


N1. Do you experience any physical stress due to physical pain or discomfort? 
N2. Do you experience mental stress? 


4.2 Selection of Features and Baseline Methods 


The proposed method uses per-activity biometric features of all activities per- 
formed in 24h before the questionnaire. In this evaluation experiment, two other 
methods (baseline 1 and baseline 2) are set as baselines. 

The common basic features used for all three methods are as follows: 


— Average RRI value in the last 24h 

— Standard deviation of RRI in the last 24h 

— Average RRI for 3h after waking up 

— Standard deviation of RRI for 3h after waking up 


'The specific features used for each method are as follows: 


Baseline 1 Lorenz plot area in the last 24h 

Baseline 2 Lorenz plot area and sleep time in the last 24h 

Proposed method Lorenz plot area for each activity (up to 6 types), Lorenz 
plot area for each activity in 3h after waking up (up to 6 types) 


4.3 Model Construction and Validation 


We use random forests to construct stress level prediction. This time, we divided 
the data into three subsets and used a three-fold cross-validation method for 
model training and validation. In constructing models, the answers (in 5 level 
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Fig. 4. Confusion matrix for the N1 (physical stress) questionnaire at night. 


likert scale) of the questionnaire were reorganized into three levels: good, bad, 
and neither good nor bad. 

To confirm the effectiveness of the per-activity biometric features, we com- 
pared the baseline method 1 (using 24-h RRI variance and Lorenz plot area as 
features), the baseline method 2 (adding sleep time as a feature to the baseline 
method 1), and the proposed method (using RRI variance and Lorenz plot area 
for each type of activities performed in 24h as features). 


5 Results 


5.1 Effects of Different Features 


The results of predicting the questionnaire results of three methods are shown 
in Table3. The proposed method achieved prediction accuracy of 0.57, 0.56, 
0.63, 0.57 for questions M1, M2, N1, and N2, respectively while baseline 1/2 did 
0.32/0.47, 0.35/0.49, 0.33/0.54, and 0.38/0.39, respectively. This suggests that 
adding sleep time features (baseline 2) improves accuracy so some extent but 
adding per-activity biometric features (the proposed method) greatly improves 
the accuracy compared to baseline 1. 


5.2 Stress Estimation Model Construction 


The stress level prediction model is constructed using random forest using fea- 
tures described in Sect.4.2. Each question’s answer obtained as stress state is 
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Table 3. Prediction accuracy of three methods for each question 


Baseline Method 1 | Baseline Method 2 | Proposed Method 
M1 | 0.32 0.47 0.57 
M2 | 0.35 0.49 0.56 
N1 | 0.33 0.54 0.63 
N2 | 0.38 0.39 0.57 


Table 4. Evaluation of recall and f-measure for questionnaire N1 


baseline method1 baseline method2 proposed method 
accuracy | 33% accuracy | 54% accuracy | 63% 

recall | f-measure recall | f-measure recall | f-measure 
Bad 0.34 | 0.37 Bad 0.53 | 0.59 Bad 0.63 | 0.69 
Good 0.35 | 0.40 Good 0.60 | 0.64 Good 0.76 | 0.70 


Table 5. Importance of nighttime questionnaire N1 (physical stress). 


Feature Importance 
RRIMean 0.119 
Lorenz plot .Bathing 0.114 
RRIMean.3hour 0.112 
RRIStd 0.093 
Lorenz plot. Goingout 0.076 
Lorenz plot. Sleeping 0.073 
Lorenz plot Eating 0.073 
Lorenz plot. Other 3hour 0.065 
Lorenz plot. Other 0.060 
Lorenz plot.Cooking.3hour | 0.052 
Lorenz plot. Cooking 0.048 
RRIStd.3hour 0.046 
Lorenz plot Eating 3hour 0.044 
Lorenz plot. Goingout.3hour | 0.026 


used as the ground truth data. The stress state data is rated on a 5-point scale, 
but since it is only necessary to judge whether the stress state is good or bad, 
the scale is reorganized to 3 levels: good, neutral, and bad. A three-fold cross- 
validation method is used to validate the model. 

We show more detailed results of N1 prediction in Table 4 and the confusion 
matrix in Fig. 4. 
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5.3 Evaluation of Feature Importance 


In order to identify the features that contribute to the estimation, we investi- 
gated their importance. Table5 shows the contribution of features in the model 
to predict N1’s answer. In Table5, the mean value and standard deviation of 
RRI had the high contribution (1st and 4th rank). Lorenz plot areas for some 
activities also show high contribution. Especially, Lorenz plot area for bathing 
activity is ranked in the second. We also confirmed that the Lorenz plot area 
of bathing often ranked high in other questionnaires. Therefore, it is suggested 
that the stress prediction accuracy could be improved with per-activity stress 
indicators more finely than just using biometric features. Also, it is suggested 
that the Lorenz plot during bathing is somehow related to stress. 


6 Conclusion 


In this paper, to improve the prediction accuracy of stress estimation, we pro- 
posed a method to predict stress level by linking daily living activities data 
and biometric data. Through the experiment using the dataset collected from 
elderly households, it is shown that the proposed method with per-activity stress 
indicators features in the last 24h improves the prediction accuracy to a great 
extent compared to the baseline methods using only stress indicators without 
distinguishing activities. 

As future work, we plan to apply the results of stress estimation obtained 
in this study to create a system that can predict QoL with high frequency and 
provide feedback on daily living activities based on the results of QOL estima- 
tion, aiming to build a system that encourages changes of activities to improve 
health state. 
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Abstract. This study is preliminary research to develop a smart cradle for women 
with spinal cord injury. The purpose of this study was to investigate the needs for 
improvement of the product and important factors related to product development. 
A focus group interview was conducted with a total of 5 women with spinal cord 
injury who had experienced parenting after spinal cord injury. After recording all 
of the focus group interviews, researchers individually analyzed the content and 
integrated the results. Easy access cradle design for wheelchair users, attachment 
of wheelchair and cradle when moving at home, an open and lockable door one 
side of the cradle were required in cradle structures. Electronic height adjustment, 
bounce mode, children's motion sensor, and function linked with a smartphone 
should be reflected in the development of the cradle. This result is meaningful in 
that it suggests points to be considered in the process of developing an assistive 
device by reflecting the desire to understand the grievance women with spinal cord 
injury when parenting. 


Keywords: Smart cradle - Spinal cord injury - Child care - Parenting - Focus 
group interview 


1 Introduction 


Parenting is one of the greatest challenges and joys adults can experience, and adults 
with spinal cord injury (SCI) who are physiologically capable of having children are no 
exception [1]. Women with SCI face various challenges in the process of conceiving, 
giving birth, and raising a child due to physical limitations [2, 3]. Mothers with SCI 
say they often see themselves as superhuman by performing parenting tasks that others 
would not be able to do because of their disability, or performing childcare tasks that 
are considered inappropriate compared to mothers with physical abilities [3—6]. Fears 
of being branded as inappropriate parents or of 'parenting' their children put some 
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parents under pressure to handle all parenting activities on their own, without the help of 
family or other support providers [7—11]. The use of assistive devices is a representative 
method that can help people with physical disabilities to solve difficulties and problems 
in the process of parenting. Assistive technology approaches such as assistive devices 
and social support play an important role in independently performing daily activities 
related to parenting. For example, a cradle designed for the physically disabled people 
can be used to care for child in the early stage of parenting, and has a great impact on the 
safety of a child. Therefore, the need for research on the development and effectiveness 
of assistive tools that are more ergonomic and functional and incorporate the latest 
technologies such as IoT systems has been raised. Along with this trend, the need for 
research that can elicit social support including financial support so that people with 
disabilities can practically use cutting-edge tools is also being emphasized [12]. The 
purpose of this qualitative study is to identify the needs related to the cradle of women 
with spinal cord injury, and to develop a smart cradle by reflecting the research results. 
Furthermore, it is expected to provide data that can elicit social support for assistive 
technology through the results of tracking research on the effectiveness, importance, 
and user satisfaction of the developed smart cradle. 


2 Method 


2. Focus Group Interview (FGI) 


In this study, a Focus Group Interview (FGI) was conducted to in-depth exploration of 
users’ needs. FGI was developed based on the method described by Krueger and Casey 
[13]. It is a qualitative research method that allows you to better understand the feelings 
and thoughts of the participants by forming a focus group with common characteristics, 
such as a study and procedure [14]. This study was conducted to reflect the considerations 
when developing the cradle by analyzing the experiences of women who have raised a 
child after spinal cord injury. 


2.2 Participants 


For one month in June 2021, research participants were recruited with help of the spinal 
cord injury association. The criteria for selection of participants in this study are as 
follows. First, women with spinal cord disorders who had experience raising a child after 
childbirth, second, those who were able to express their opinions sufficiently, and third, 
those who agreed to participate in the study after hearing the explanation of the purpose 
and method of the study. Total 5 subjects who met the selection criteria participated in 
this study (Table 1). 


2.3 Research Questions 


The focus group interview questionnaire used in this study was composed of open, 
introductory, key, and closing questions based on the method suggested by Kruger and 
Casey. The key questions of this interview were as follows: (Q1) have you ever used 
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Table 1. General characteristics of participants (n = 5) 


No Dx Type of impairment Age Duration of disability 
(complete/incomplete) (years) (years) 

A SCI diplegia Complete 40 19 

B SCI diplegia Complete 58 21 

C SCI diplegia Incomplete 52 30 

D SCI diplegia Incomplete 51 25 

E SCI diplegia Incomplete 41 14 


a cradle? (Q2) what were the inconveniences when using a regular cradle? (Q3) how 
did you put your child on the bed?, how did you hold your child out of bed? (Q4) what 
aspects of the cradle would you like to improve considering your physical disabilities? 


2.4 Data Collection & Analysis 


We informed study participants about the interview contents and process before FGI. 
Interview was conducted through videoconference and participants feely expressed their 
experiences and thoughts for 2 h. All data were collected by recording interview video. 
Researchers categorized the recorded contents according to common themes using the 
technique suggested by Krueger [15]. Similar sentences, experiences were classified, and 
data were analyzed by grouping important themes and categories through continuous 
review. This qualitative research method focuses on revealing the essence of experience 
in the context of the participant’s situations. 


3 Results 


3.1 Needs for Cradle Structures 


Various needs were investigated for the structural design of the cradle desired by the 
participants in order to consider the efficiency of the cradle development. SCI patients 
spend most of their time at home in wheelchairs. Therefore, in order to take care of 
a child, lower part of the cradle must be manufactured so that there is a space for a 
wheelchair to enter. They also said that wheelchair should be able to move with a cradle 
attached (Fig. 1), there should be an open and lockable door one side of the cradle to 
pull a child up (Fig. 2). 

“A wheelchair must be able to fit under the cradle because we need to take care of 
a child as close as possible" (Participant E). 

"It would be nice if the wheelchair could fit in the cradle and move together" 
(Participant B). 

"Cradle should have an open and lockable door for a child's safety and ease of use" 
(Participant A). 

"Need a storage space around the cradle to put children's items" (Participant C). 
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P 
Fig. 1. Attachment of cradle & wheelchair Fig. 2. Open and lockable door on cradle 


3.2 Needs for Cradle Functions 


Participants said that they needed an electronic height adjustment function to change the 
position of the cradle. Motion sensor and a bouncing mode are needed to detect child's 
movements (Fig. 3). Finally, they argued that the development of a smart cradle based 
on artificial intelligence that can be linked with a smartphone is necessary to monitor 
child's health and surrounding environment (Fig. 4). 

"Need a lift function that can adjust height of the cradle by pressing switch" 
(Participant C). 

“Tt is necessary to be able to control the cradle and check the condition of a child 
with a smart phone." (Participant D). 

“The electric bouncing function seems to be useful for caring for a child" (Participant 
B). 


Fig. 3. Bouncing function using smartphone Fig. 4. Monitoring child's health & 
environment 


4 Conclusion 


We tried to find out important factors related to product development. A focus group 
interview was conducted with 5 women with spinal cord injury, and various opinions 
were derived. In terms of design and structure, it is important to easily access the cradle 
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in a wheelchair and attach the cradle to the wheelchair when moving. they also wanted 
to have a door that could be opened and locked on one side. Electric height adjustment 
and bouncing, as well as checking the health and environment of the child with a smart 
phone are necessary for the function. This study is meaningful in that it presented the 
difficulties that disabled with SCI may experience during childrearing as a factor to be 
considered in the development of assistive devices. It is hoped that it will be helpful in 
the development of parenting devices for the disabled people. 


Acknowledgments. The research was supported by a grant of the Korea Health Technology R&D 
Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry 
of Health & Welfare, Republic of Korea (grant number: HJ21C0014). 


References 


1. Kaiser, A., Reid, D., Boschen, K.A.: Experiences of parents with spinal cord injury. Sex 
Disabil. 30(2), 123-137 (2012) 

2. Preston, P., Jakobson, M.: Keeping our Families Together: A Report of the National Task 
Force on Parents with Disabilities and their Families. Through the Looking Glass, Oakland 
(1997) 

3. McKeever, P., Angus, J., Miller, K.L., Reid, D.: It's more of a production: accomplishing 
mothering using a mobility device. Disabil. Soc. 18, 179—197 (2003) 

4. Carty, E.M.: Disability and childbirth: meeting the challenges. CMAJ 159(4), 363—369 (1998) 

5. Prilleltensky, O.: A ramp to motherhood: the experiences of mothers with physical disabilities. 
Sex Disabil. 21(1), 21-47 (2003) 

6. Shein, F., Nantais, T., Tonack, M.: An Information Resource Describing Consumer Experi- 
ences with Assistive Technology: Spinal Cord Injury Peer Information Library on Technology. 
Toronto Rehabilitation Institute, Toronto (2003) 

7. Kirshbaum, M.: A disability culture perspective on early intervention with parents with 
physical or cognitive disabilities and their infants. Infants Young Child 13(2), 9-20 (2000) 

8. Frontera, W.R.: Delisa's Physical Medicine & Rehabilitation: Principles and Practice, 5th 
edn. Lippincott Williams & Wilkins, Philadelphia (2010) 

9. Prilleltensky, O.: My child is not my carer: mothers with physical disabilities and the well- 
being of children. Disabil. Soc. 19(3), 209—223 (2004) 

10. Tulega, C., DeMoss, A.: Babycare assistive technology. Technol. Disabil. 11, 71—78 (1999) 

11. Olkin, R.: Are children of disabled parents at risk for parentification? Parent. Disabil. 8(2), 
8—9 (2000) 

12. Gitlow, L., et al.: Parenting with a disability: exploring the satisfaction and efficacy of the 
PediaLift wheelchair accessible crib. Disabil. Rehabilitation: Assist. Technol. 16(8), 902—909 
(2021) 

13. Krueger, R.A., Casey, M.A.: Focus Group: A Practical Guide for Applied Research. Sage, 
LosAngeles, CA (2009) 

14. Krueger, R.A.: Focus Groups: A Practical Guide for Applied Research. Sage publications 
(2014) 

15. Krueger, R.A.: Analyzing and Reporting Focus Group Results, vol. 6. Sage Publications, 
2455 Teller Road, Thousand Oaks California 91320 United States (1998) 


216 J. Kim et al. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


® 


Check for 
updates 


ICT-Based Customized Off-Loading Cushion 
to Prevent Pressure Ulcers for People 
with Spinal Cord Injury: A Pilot Study 


Yun-hwan Lee! C3), Kwang-tae Moon!, Dong-wan Kim!, and Jongbae Kim2€9 


l Department of Occupational Thearpy, The Graduate School, Yonsei University, Seoul, 
South Korea 
enn1210@gmail.com, otbuilder@yonsei.ac.kr, dwan3303@naver.com 
2 Department of Occupational Thearpy, College of S/W Digital Healthcare Convergence, Yonsei 
University, Wonju, South Korea 
jongbae@yonsei.ac.kr 


Abstract. The wheelchair cushion is one of the intervention methods for pre- 
venting pressure ulcers in people with spinal cord injuries. Recently, a customized 
wheelchair cushion along with off-loading technology that distributes pressure 
by removing the seat surface in contact with the bony protrusion of the buttocks 
in the sitting position is attracting attention. In spite of this, they are exposed to 
the risk of secondary pressure ulcers because they cannot recognize their body 
pressure distribution and unintended posture changes due to sensory dysfunction. 
Accordingly, we developed an ICT-based off-loading cushion that can monitor 
the pressure distribution and pelvis alignment in a sitting position in real-time. 
People with spinal cord injuries who participated in the pilot study were satisfied 
with the device and service, and the ICT-based customized off-loading cushion 
had a positive psychosocial effect on them. We hope that this will contribute to 
improving the quality of life by preventing pressure ulcers in people with spinal 
cord injuries. 


Keywords: ICT-based off-loading cushion - Pressure ulcers - Spinal cord injury 


1 Introduction 


Pressure ulcers is local injury that occur to the skin and internal tissues of bony pro- 
trusions due to excessive pressure and shear force [1]. It requires lots of cost and time 
for treatment and negatively affects a wide range of areas such as functional activities, 
psychosocial factors, participation, and quality of life of people with spinal cord injuries 
[2, 3]. 

The wheelchair cushion is representative intervention for preventing pressure ulcers 
in people with spinal cord injuries. The wheelchair cushion can prevent skin damage by 
dispersing pressure of buttocks in sitting position [4]. Itis largely divided into traditional 
technologies in the form of air, gel, and foam, and new technologies that combined 
various materials and off-loading technologies [5]. 
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People with spinal cord injuries are restricted from sensory function below the dam- 
aged neurologic segment [6, 7]. So, even though the wheelchair cushion is used, excessive 
pressure and shear force in the buttocks caused by using powered seating functions and 
outdoor activities are not recognized by themselves. Due to this, people with spinal cord 
disabilities are exposed risk of secondary pressure ulcers. 

Therefore, this study aims to develop an ICT-based system that can monitor pressure 
distribution and posture changes in real-time when people with spinal cord disabilities 
use wheelchairs, and to investigate their satisfaction and psychosocial effects. 


2 Method 


2.1 Participants 


The participants of this study were three people with spinal cord injuries living in the 
community. All of the participants were male, and the average age was 43.7 years. 
They are complete cervical level injuries (C3,4 and 6) and the average duration of the 
onset is 15.3 years. Two participants use power wheelchairs and the other uses manual 
wheelchair. Their average daily use of wheelchairs is 8.3 h (Table 1). 


Table 1. Characteristics of participants 


Characteristics Participant 

A B C 
Gender Male Male Male 
Age 53 43 35 
Diagnosis C4 ASIA A C3 ASIA A C6 ASIA A 
Onset (Year) 26 10 10 
Type of wheelchair Power Power Manual 
Wheelchair use (hour/for a day) 5 10 10 


2.2 ICT-Based Customized Off-Loading Cushion 
2.2.1 Customized Off-Loading Cushion 


The off-loading cushion is a wheelchair cushion with off-loading technology to prevent 
pressure ulcers. Off-loading technology is distributing the pressure by removing the 
seat surface and adjusting the height in contact with the main bony protrusion of the 
buttocks (Ischial Tuberosity, Coccyx and Greater Trochanter) in a sitting position [8]. It 
is contoured to fit the shape of the buttocks, so it is possible to maintain stability when 
seated in a wheelchair and minimize shear force (Fig. 1). 

The off-loading cushion was customized through clinical and manufacturing phase. 
In clinical phase, occupational therapist evaluates the clinical feature and the wheelchair 
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Greater Trochanter Coccyx 


Ischial Tuberosity 


Greater Trochanter 


Fig. 1. Off-loading cushion 


being used. After the evaluation, we captured and scanned a sitting posture using a 
vacuum bag and 3D scanner. Next, we performed 3D modeling through CAD (Computer 
Aided Design) programs to fit each user’s buttocks. And we adjusted the seat surface 
to minimize the pressure of the buttocks based on the result of clinical evaluation. In 
manufacturing phase, we performed a simulation process using the CAM (Computer 
Aided Manufacturing) software. After that, EVA (Ethylene-Vinyl Acetate) foam was 
processed into a cushion using a CNC (Computer Numerical Control) machine (Fig. 2). 


Clinical Phase 


Capturing 
E Demie (aD) 
wo f 


3D Modeling 


” D 


Collect Needs and Information Vacuum bag 3D Scanner CAD 


Manufacturing Phase 


Manufacturing 


x hd 


CAM EVA Foam, CNC Off loading cushion. 


T 


Fig. 2. Manufacturing process of customized off-loading cushion 


2.2.2 Pressure Distribution and Posture Change Monitoring System 


Monitoring system is divided into hardware and software. The hardware of system 
consists of sensors, BLE (Bluetooth Low Energy), and a multiplexer. A total of 10 
sensors were attached based on the coccyx, greater trochanter, and ischial tuberosity, 
which are the most common areas for pressure ulcers in sitting position (Fig. 3). 

Three sensors were attached 2—3 cm below the coccyx and four on the slope between 
the ischial tuberosity and the thigh, and one on the ischial tuberosity area to detect the 
pelvis forward sliding. In addition, two sensors were attached to the greater trochanter 
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nave 


Fig. 3. The hardware of monitoring system 


to detect the sideways sliding of the pelvis. The hardware was placed on the bottom of 
the off-loading cushion using the case. 

The data input through the sensor is transmitted to the monitoring application, which 
is the software of the system. The application can be linked with the user’s tablet or 
smartphone. Users can intuitively check the pressure distribution and the alignment of 
pelvis through changes in pressure values and colors. The degree of abnormal posture 
can be known through the icons. It also provides immediate feedback on posture status 
to users by providing notifications through banners on their smartphones or tablets when 
abnormal posture is detected (Fig. 4). 


Fig. 4. The software of monitoring system (monitoring app.) 


2.3 Study Protocol 


Participants performed activities of daily living and community activities for a week with 
ICT-base customized off-loading cushions. After that, we were measured satisfaction 
and psychosocial effect on ICT-based customized off-loading cushions. 
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2.3.1 Satisfaction 


Satisfaction on ICT-based customized off-loading cushions was measured through K- 
QUEST 2.0 (Korea-Quebec User Evaluation of Satisfaction Assistive Technology 2.0). 
This is a 12-item related to the satisfaction on assistive technology device (8 items) and 
assistive technology service (4 items). Each item is scored on a 5-point Likert scale 
ranging from 0 to 5, and a higher score means higher satisfaction [9]. 


2.3.2 Psychosocial Impact 


The psychosocial impact of ICT-based customized off-loading cushions was measured 
using K-PIADS (Korea-Psychosocial Impact of Assistive Devices Scales). This is a 26- 
item, self-report questionnaire and consist of three sub-scale: Competence (12 items), 
Adaptability (6 items), and Self-esteem (8 items). Each item is scored on a 7-point Likert 
scale ranging from —3 to 3. A positive score mean that assistive technology device has 
a positive psychosocial impact on user [10]. 


3 Result 


3.1 Satisfaction 


The average for the mean score of the assistive technology device was 4.6. The mean 
score of comfort and effectiveness was the highest, and the mean score of weight and 
durability was the lowest (Table 2). 


Table 2. The result of satisfaction on assistive technology device 


Item Participants 
A B C Mean 

Dimension 4 5 5 4.7 
Weight 3 4 5 4.0 
Adjustment 4 5 4 4.3 
Safety 4 5 5 4.7 
Durability 4 5 3 4.0 
Easy to use 4 5 5 4.7 
Comfort 5 5 5 5.0 
Effectiveness 5 5 5 5.0 


The average for the mean score of the assistive technology service was 4.5. The mean 
score of service delivery and professional service was the highest, and the mean score 
of repair and follow-up service was the lowest (Table 3). 
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Table 3. The result of satisfaction on assistive technology service 


Item Participants 

A B C Mean 
Service delivery 4 5 5 4.7 
Repair service 3 5 5 4.3 
Professional service 5 5 4 4.7 
Follow-up service 3 5 5 4.3 


3.2 Psychosocial Impact 


The mean scores for competence, adaptability, and self-esteem were 2.39, 2.61, and 2.42, 
respectively. it means that the ICT-based customized off-loading cushion had a positive 
to very positive impact on the participants. (Table 4). 


Table 4. The result of psychosocial impact 


Scale Participants 

A B c Mean 
Competence 2.17 2.42 2.58 2.39 
Adaptability 2.50 12.67 2.67 2.61 
Self-esteem 2.13 2.63 2.50 242 


4 Conclusion 


People with spinal cord injuries suffer from permanent physical and sensory dysfunction 
due to trauma or disease. Therefore, intervention strategies should focus on participation 
and improvement of quality of life rather than functional recovery. For a healthy life for 
people with spinal cord disabilities, it is necessary to actively cope with complications 
such as pressure ulcers that reduce the quality of life. A wheelchair cushion is a technical 
intervention to prevent pressure ulcers. In this study, we developed a wheelchair cushion 
applying customized off-loading technology and ICT technology that can monitor body 
pressure distribution and posture change. As a result of applying it to three people with 
spinal cord disabilities living in the local community for a week, satisfaction with the 
ICT-based off-loading cushion was appropriate, and it had a positive psychosocial effect. 
Through this pilot study, we derived the need for a covering to protect hardware from 
urinary incontinence and improvement in applications such as colors that can intuitively 
recognize pressure distribution during outdoor activities. Currently, we are supplement- 
ing software and hardware based on the results derived. In the future, we intend to 
investigate the effects of ICT-based off-loading cushions on the clinical effectiveness, 
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quality of life and participation through a randomized controlled trial study for people 
with spinal cord disabilities living in local communities. This study has a limitation in 
that it was conducted with only three male subjects. Therefore, we intend to include 
balanced gender subjects with an appropriate number of sampling to test for statistical 
significance in future study. We expect this to be useful in improving the quality of life 
for people with spinal cord injuries as a health care system. 
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Abstract. Some diseases are characterized by persistent deficits in brain activity. 
Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder. It appears 
in early childhood and evolves throughout life and needs to be detected early 
to accelerate the treatment and recovery process. These deficits may be detected 
using medical imaging techniques. In this paper, we present machine learning 
algorithms allowing to detect peoples with ASD from normal peoples. We used 
data from the ABIDE dataset. We tested 3 algorithms: Support Vector Machines 
(SVM), Long Short-Term Memory (LSTM) and Convolutional Neural Network 
(CNN). The best result was obtained using CNN algorithm with an accuracy equal 
to 95%. 


Keywords: Autism - Machine learning - CNN - LSTM - SVM - ABIDE - fMRI 


1 Introduction 


In the recent decades, the medicine industry has adopted numerous approaches and 
methods to discover or predict many diseases [1, 2]. In this context, researches are 
focusing on developing machine learning algorithms in order to improve their accuracy 
[3]. 

Brain diseases such as Autism Spectrum Disorder (ASD) have been of increasing 
interest to researchers over the past few years. ASD is known as a cerebral disease that 
involves impairments in cognitive functions, communication/social interactive, cognitive 
and adaptive skills. Studies from the neuroscience domain indicate that the biomarkers 
of ASD are still unknown however the corpus callosum and intracranial brain volume 
holds significant information for its detection. Based on these conclusions, we suggested 
machine learning models for automatic ASD detection. The proposed algorithms were 
tested and evaluated on the ABIDE! dataset. Since autism is a brain dysfunction disorder, 
we will use functional Magnetic Resonance Images (fMRI) which better describe this 
disorder. 

In this paper three algorithms were presented: Support Vector Machine (SVM), 
Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN). The 
comparison of results showed that CNN performs better than SVM and LSTM with an 
accuracy equal to 95%. 


! ABIDE dataset: https://fcon_1000.projects.nitrc.org/indi/abide/. 
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2 State of Art 


ASD is a brain-based disorder marked by social deficits and repetitive behaviors. The 
main aim of psychiatric Neuroimaging research is to identify objective bio-markers that 
may inform the diagnosis and treatment of brain-based disorders. ASD is associated 
with a range of phenotype that vary in severity of social, communicative and sensory 
motor deficits. Recently, Machine Learning (ML) algorithms have been applied to brain 
imaging data to extract brain function patterns. These algorithms can retrieve robust 
neural patterns from brain imaging data of psychiatric disorder patients. 

In their work, Omar et al. [4] proposed an effective prediction model based on ML 
technics in order to develop a mobile application for predicting ASD for people of any 
age. The model was developed by merging Random Forest-CART and Random Forest- 
Iterative Dichotomiser3. The proposed model was evaluated with AQ-10 dataset and 
250 real dataset collected from people with and without autistic traits [4]. 

In another work, Kuper et al. [5] focused on adolescents and adults. They used SVM 
to examine whether ASD detection can be improved by identifying a subset of behav- 
ioral features from the ADOS Module 4 in a routine clinical sample. They identified 
reduced subsets of 5 behavioral features for the whole sample as well as age subgroups. 
The results of evaluation may help to improve the complicated diagnostic process of 
ASD by encouraging future efforts to develop novel diagnostic instruments for ASD 
detection based on the identified constructs as well as aiding clinicians in the difficult 
question of differential diagnosis [5]. In the work of [6], authors explore the possibility 
to test Naive Bayes, Support Vector Machine, Logistic Regression, KNN, Neural Net- 
work and Convolutional Neural Network for predicting and analysis of ASD problems. 
They evaluate the proposed methods on publicly available three different non-clinically 
ASD datasets for children, adolescents and adults. The results suggest that CNN based 
prediction models work better on all these datasets. 

Knowing that the Functional Magnetic Resonance Imaging (fMRI) helped to identify 
and detect the ASD, Dvornek et al. [7] proposed a model based on Recurrent Neural 
Networks with Long Short-Term Memory for classification of individuals with ASD and 
typical controls from the resting-state fMRI time-series. They used the ABIDE-I dataset 
for training and testing the LSTM models. Results with a cross-validation framework 
showed an accuracy of 68.5% on the whole ABIDE cohort. 

The primary goal of the current study is to classify ASD patients and control par- 
ticipants based on their neural patterns of functional connectivity using resting state 
functional magnetic resonance imaging (rs-f[MRI) data. Supervised ML algorithms were 
used and applied to a large population sample of brain imaging data extracted from the 
ABIDE dataset. 
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3 Methods 


3.1 Convolutional Neural Network (CNN) 


The Convolutional Neural Network (CNN) is a deep learning neural network that has 
shown excellent performance in many computer vision and machine learning prob- 
lems. It is designed to learn automatically spatial hierarchies of features through back- 
propagation. It uses the multiple building blocks, such as the convolution layers, the 
pooling layers and the fully connected layers [8]: 


Convolution Layer. This layer is a fundamental component of the CNN architecture. 
It performs feature extraction using a combination of linear and nonlinear operations, 
i.e. convolution operation and activation function. 


Pooling Layer. This layer provides a down sampling operation which reduces the 
dimensional of the feature maps and identify invariance to translation, shift and distor- 
tions. The filter size, stride and padding are the hyper-parameters in pooling operations, 
similar to convolution operations. 


Fully Connected Layer. The features extracted by the convolution layers and then 
downsampled by the pooling layers are mapped using a subset of fully connected layers 
to the final outputs of the network. The fully connected layer is followed by a nonlinear 
function, such as ReLU. When the input data are transformed into output through the 
different layers, it is called the forward propagation. We build a CNN network with 8 
convolution layers, 4 pooling layers and a fully connected layer. 


3.2 Long Short Term Memory (LSTM) 


Long Short Term Memory (LSTM) network is a special kind of Recurrent Neural Net- 
works that is capable of learning long-term dependencies. LSTM can be used to model 
univariate time series forecasting problems. These problems are comprised of a single 
series of observations and a model is required to learn from the series of past observations 
to predict the next value in the sequence. The LSTM model learn a function that maps a 
sequence of past observations as input to an output observation. As such, the sequence 
of observations must be transformed into multiple examples from which the LSTM can 
learn [7]. 

In our paper, we extracted mean time-series from regions of interest defined by 
several atlases. Each time course was normalized to represent percent change from the 
average signal for that region of interest. 


3.3 Support Vector Machines (SVM) 


Support Vector Machine (SVM) is a classical binary classification algorithm, particularly 
adapted for big data. The SVM method was proposed by “Vapnik” [9] and has been 
widely used for medical image processing. 
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Given a set of training examples features (ulus and their labels (yk, a Support 
Vector Machine (SVM) builds a prediction model that assigns new examples into one 
class or the other depending on the side of the learned separating hyperplan on which 
they stand. 

The decision function of the SVM is given in (1) where k is a given positive kernel. 


l 
fœ = Y  afk(x, x). (1) 
j=l 


The dual learning optimization problem form is given in (2). C>= 0 is the 
regularization parameter related to the misclassified samples. 


I I 
1 
max 5,a; — 5 2, aiajyiyjk i, xj), 
1 


9i j=] ij= 
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The linear kernel was used and the one-vs-the-rest scheme was adopted in our work. 
For dimensionality reduction we used PCA to keep 99% of variance then we transformed 
the phenotypic data into a dataset and determined the target. 


4 Material and Results 


4.1 Material 


This study is performed using the ABIDE dataset which is an online sharing consortium 
that provides imaging data of ASD and control participants with their phenotypic infor- 
mation. The ABIDE dataset includes two large-scale collections: ABIDE I and ABIDE 
II. ABIDE I includes 1112 subjects aged between 7 and 64: 539 subjects with ASD and 
573 subjects with typical controls. 

The ABIDE dataset includes functional and structural brain imaging data collected 
from laboratories around the world to ensure data diversity and to understand the com- 
plexity of the disease to be able to make the diagnosis at earlier ages, to select optimal 
treatments and to predict outcomes. 

Structural imaging [9] is used to visualize and analyze anatomical properties of the 
brain and are particularly useful for detecting brain damage and abnormalities. Func- 
tional imaging [9-11] is used to identify brain areas and underlying brain processes that 
are associated with performing a particular cognitive or behavioral task. It can be used 
to investigate the functional anatomy of the brain by identifying the parts of the brain 
with critical functions, to evaluate the effect of many diseases such as stroke and to brain 
treatment. 

Our imaging data is composed of functional Magnetic Resonance Imaging from 
ABIDE I that allows a better description of the brain-related dysfunction. We extracted 
features using each correlation matrix extracted from the dataset. For dimensionality 
reduction, we used a PCA to keep 99% of variance. 

An example of F-MRI images is shown in Fig. 1. 
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Fig. 1. Example of F-MRI images 


Our proposed algorithms have been tested on a dataset of 500 patients. 51.2% of the 
base are subjects with ASD while 48.8% are subjects with typical controls (Fig. 2). We 
can say that our dataset is balanced. We assign 70% of our modeling dataset to the train 
and the remaining 30% to the test. 


contro! 


asd 


Fig. 2. Diagram of ASD distribution in the ABIDE dataset 


For gender distribution (Fig. 3), we can see that our dataset is mainly composed 
of males with a rate = 84%. To understand the reason behind this, we need to know 
that ASD affects females less frequently than males. Of course several sex-differential 
genetic and hormonal factors may contribute in the appearance and the development of 
this disease. 

While visualizing the age distribution (Fig. 4), it’s clear that the most of patients are 
very young (aged from 5 to 15 years). This may be justified by the fact that this disease 
is more frequent among children than among adults and emphasizes the importance of 
early ASD diagnosis for children to accelerate treatment and recovery. 


230 N. Boughattas and H. Jabnoun 


Female 


Male 
Fig. 3. Diagram of gender distribution in the ABIDE dataset 
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Fig. 4. Age distribution in ABIDE dataset 


4.2 Results and Discussion 
The performance of the proposed methods has been evaluated using the accuracy score. 


LSTM. We used the binary cross-entropy loss function and the Adadelta optimizer 
with the default parameter values. During the training, we fixed the dropout rate during 
training to 0.4. The impact of parameters and variations of the proposed architecture were 
explored as well as training conditions. We tested the data while varying the number 
of hidden nodes (8, 16, 32, or 64) in the LSTM, and removing dropout. We also tested 
variations on the base network: connecting only the final LSTM cell's output to a single 
dense node and stacking LSTM layers. The variation of Loss and Accuracy rates are 
presented in Fig. 5. We got an accuracy equal to 76.2596. 


CNN. CNN is typically a repetitions of a stack of several convolution layers and pooling 
layers, followed by one or more fully connected layers. The CNN that we have worked 
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Fig. 5. Accuracy and loss depending on the number of epoch for LSTM model 


on has 8 convolutional layers, 4 max pooling layers and one sigmoid output layer. The 
input consists of three (316, 70, 1) patches from axial, sagittal and coronal image slices 
centered on the target voxel. We used Adam as optimizer algorithm. We fit the model 
and set the number of epochs at 60, so we got an accuracy value equal to 95% (Fig. 6). 


Loss 
" NNNM on 
—— train 
—— test 
01 
T T T T T 
0 10 20 Accumacy 40 50 60 
—— train 
0.98 —— test 
0.96 
0.94 
0 10 20 30 40 50 60 


Fig. 6. Accuracy and loss depending on the number of epoch for CNN model 


SVM. The obtained results show good performance of our SVM based method with an 
accuracy rate equal to 88%. The feature reduction step based on PCA allowed to keep 
99% of variance. Our results exceed those of Eslami et al. [12] who combined SVM with 
a deep learning algorithm. We can say also that our SVM based method performs better 
than the method presented in [13] for male subjects since our base is male oriented. 
The Linear SVM method presented in [14] reaches an accuracy equal to 69% and the 
Gaussian SVM method presented in [15] reached 66%. These two methods are less 
efficient than our method. 

In the Table 1, we summarize the obtained results for the trained algorithms. We 
conclude that CNN gives the best accuracy rate. 
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Table 1. Accuracy rate of ASD detection in ABIDE dataset using different algorithms. 


Algorithm Accuracy (%) 
CNN 95 

SVM + PCA 88 

LSTM 76.25 

SVM + Deep learning [12] 80 

SVM + GARCH [13] 71 

Linear SVM [14] 69 

Gaussian SVM [15] 66 


5 Conclusion 


ASD is a brain disease that it is notoriously difficult to diagnose, especially for children. 
It is associated with significant brain structure changes, which can be measured by 
magnetic resonance imaging (MRI) scan. 

In this paper, we presented three algorithms to generate a model for ASD detec- 
tion using image classification tools: Convolutional Neural Network, Support Vector 
Machines and Long Short Term Memory. 

These methods for automatic ASD detection were tested on the ABIDE dataset. 
The performance of algorithms was evaluated throw the accuracy. The best result was 
obtained using the CNN algorithm with a rate equal to 9596. 

In order to achieve better recognition of ASD, other modalities i.e. EEG, speech or 
kinesthetic modalities can be analyzed simultaneously. 


References 


]. Uddin, S., Khan, A., Hossain, M.E., Moni, M.A.: Comparing different supervised machine 
learning algorithms for disease prediction. BMC Med. Inform. Decis. Mak. 19, 281 (2019) 

2. Dahiwade, D., Patle, G., and Meshram, E.: Designing disease prediction model using machine 
learning approach. In: 2019 3rd Inter-national Conference on Computing Methodologies and 
Communication (ICCMC), pp. 1211-1215 (2019) 

3. Marcos-Zambrano, L.J., et al.: Applications of machine learning in human microbiome stud- 
les: a review on feature selection, biomarker identification, disease prediction and treatment. 
Front. Microbiol. 12, 313 (2021) 

4. Omar, K.S., Mondal, P., Shahnaz Khan, N., Karim Rizvi, M.R., Islam, M.N.: A machine learn- 
ing approach to predict autism spectrum disorder. In: International Conference on Electrical, 
Computer and Communication Engineering (ECCE) (2019) 

5. K'upper, C., et al.: Identifying predictive features of autism spectrum disorders in aclinical 
sample of adolescents and adults using machine learning. Sci. Rep. 10(1) (2020) 

6. Raja, S., Masood, S.: Convolutional neural net-works: an overview and application in 
radiology. Procedia Comput. Sci. 167, 994—1004 (2020) 

7. Dvornek, N.C., Ventola, P., Pelphrey, K.A., Duncan, J.S.: Identifying autism from resting-state 
fMRI using long short-term memory networks. Mach. Learn. Med. Imaging 10541, 362-370 
(2017) 


11. 


12. 


13. 


14. 


15. 


Autism Spectrum Disorder (ASD) Detection Using Machine Learning 233 


. Yamashita, R., Nishio, M., Do, R.K.G., Togashi, K.: Convolutional neural networks: an 


overview and application in radiology. Insights into Imaging 9(4), 611—629 (2018). https:// 
doi.org/10.1007/s13244-018-0639-9 


. Vapnik, V.N.: The Nature of Statistical Learning Theory (1995) 
. Stigler, K.A., McDonald, B.C., Anand, A., Saykin, A.J., McDougle, C.J.: Structural and 


functional magnetic resonance imaging of autism spectrum disorders. Brain Res. 1380, 146— 
161 (2011) 

Glover, G.H.: Overview of functional magnetic resonance imaging. Neurosurg. Clin. 22(2), 
133-139 (2011) 

Eslami, T., Fahad S.: Auto-ASD-network: a technique based on deep learning and support 
vector machines for diagnosing autism spectrum disorder using fMRI data. In: Proceedings 
of the 10th ACM International Conference on Bioinformatics, Computational Biology and 
Health Informatics (2019) 

Sartipi, S., Mahrokh G.S., Hashem K.: Diagnosing of autism spectrum disorder based 
on GARCH variance series for rs-fMRI data. In: 2018 9th International Symposium on 
Telecommunications (IST). IEEE (2018) 

Plitt, M., Kelly A.B., Alex M.: Functional connectivity classification of autism identifies 
highly predictive brain features but falls short of biomarker standards. Neuroimage Clin. 7, 
359-366 (2015) 

Chen, C.P., et al.: Diagnostic classification of intrinsic functional connectivity highlights 
somatosensory, default mode, and visual regions in autism. Neuroimage Clin. 8, 238—245 
(2015) 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 


The images or other third party material in this chapter are included in the chapter's Creative 


Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter's Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


D) 


Check for 
updates 


Ant Colony Optimization with BrainSeg3D 
Protocol for Multiple Sclerosis Lesion Detection 


Dalenda Bouzidi! 3), Fahmi Ghozzi23), and Ahmed Fakhfakh?3 (69) 


l National Engineering School of Sfax (ENIS), Sfax University, Sfax, Tunisia 
dalendabouzidi91@gmail.com 
2 National School of Electronics and Telecommunications of Sfax (ENET' COM), Sfax 
University, Sfax, Tunisia 
(fahmi.ghozzi,ahmed.fakhfakh)Genetcom.usf.tn 
: Digital Research Center of Sfax (CRNS), Laboratory of Signals, Systems, Artificial 
Intelligence and Networks (SM GRTS), Sfax, Tunisia 


Abstract. Magnetic resonance imaging (MRI) has quickly established itself as 
the reference imaging tool for the management of patients suffering from multiple 
sclerosis (MS), both for the diagnosis and the follow-up of the evolution and 
evaluation of the impact of new therapies. 

The treatment of multiple sclerosis does not cure the disease, but it slows its 
progression and can help to space out attacks. In this paper, tumor segmentation is 
treated as a problem of classification using the Ant Colony optimization algorithm 
(ACO) combined with a proposed protocol based on BrainSeg3D tools. Many 
studies and many existing approaches tend the multiple sclerosis (MS) which is a 
chronic inflammatory anomaly of the central nervous system. 

The aim of this work is to evaluate and to verify the effectiveness of the pro- 
posed protocol on a public longitudinal database which contains 20 MS patients. 
This study is concerned with comparing these results against the ground truth 
performed by two experts and against other methods namely Dissimilarity Map 
(DM) creation and segmentation in terms of Dice Similarity Coefficient (DSC). 


Keywords: Multiple sclerosis - Segmentation - ACO BrainSeg3D - 
Longitudinal dataset - Ground truth 


1 Introduction 


During the research work, we focused on brain MRI analysis, particularly in the context 
of monitoring the condition of patients with multiple sclerosis. MRI is considered one 
of the complementary examinations in the diagnostic approach of this disease since it 
plays a key role in monitoring the evolutionary state of the disease in patients. 

The major problem lies in the difficulty of analyzing and reading these MRI images 
because of the variability in the size, the contrast and the difficulty of locating the lesions, 
and the automatic segmentation of the lesions. Several recent studies have worked on 
the automatic segmentation of MS lesions based on several approaches to follow the 


© The Author(s) 2022 
H. Aloulou et al. (Eds.): ICOST 2022, LNCS 13287, pp. 234—245, 2022. 
https://doi.org/10.1007/978-3-031-09593-1 19 


Ant Colony Optimization with BrainSeg3D Protocol for Multiple Sclerosis 235 


evolutionary state of this disease which can provide information on therapeutic efficacy. 
To detect lesions from an MRI image that carries this disease, it is necessary to adopt an 
efficient algorithm that can easily and efficiently detect the lesions. 

To solve the problem of image segmentation and in particular to improve the per- 
formance of segmentation methods by applying them for different types of images. The 
objectives we have set ourselves are to contribute to the development and optimization 
of new segmentation criteria. We adapted the metaheuristic methods by simultaneously 
applying several criteria to segment the images. As we have created a new protocol 
based on filters and automatic tools that allow us to facilitate the task of segmentation 
and detection of MS lesions. Our motivation is to detect MS lesions for different patients 
included in novel longitudinal MRI datasets. 

The residual of this study is orderly as follows: we briefly present in Sect. 2 some 
previous works; in Sect. 3 presents the proposed algorithm ACO_BrainSeg3D. Then we 
will detail the public longitudinal database we used for the test and the validation of our 
algorithm. Then Sect. 4 presents some results and discussion. Finally, we would end up 
with a conclusion and submit some perspectives. 


2 Previous Works 


The segmentation techniques can be classified according to different criteria and they 
can be presented according to a classical methodological angle. These approaches are 
then grouped into parametric, non-parametric and geometric approaches. Other states 
of the art present the different techniques by defining the classical approaches among 
recent approaches. 

For the past twenty years, and despite the appearance of other new, more sensitive 
and more advanced techniques, nuclear magnetic resonance imaging has presented the 
tool frequently used for the diagnosis of MS since it makes it possible to detect this 
pathology at an earlier stage than in the past. To solve the posed problems for diagnosis, 
segmentation of MRI images is posed as an important task for detecting MS lesions. 

In parallel with our research work for segmentation and the detection of SEP lesions, 
there are other similar works in progress essentially for the detection and quantification 
of the atrophy of the spinal cord SC in the MRI images are the key to the key. In 2018, [1] 
presented a fully automatic method “OPTIC” to detect the median cord line in the MRI 
volumes and locate the Spinal Cord (SC) in the MRI images. The authors have validated 
their method in healthy subjects and patients with very variable spatial resolutions and 
including some artifacts. The interest of this method is to detect the median line of the 
SC that can be used as initialization for other SC segmentation methods for pathological 
cases. In 2019, the authors [2] develop a fully automatic framework for segmenting SC 
and intra medullary lesions on a variety of MRI contrasts. The proposed approach is 
based on a sequence of two convening neuron networks (CNN): 


e First CNN with dilated 2D circumfolutions to detect the median line of the SC. 
e Tracking a second CNN with dilated 3D circumfolutions to segment SC and / or 
lesions. 
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In the same year, McCoy [3] used 2D convoluting neuron networks for the automatic 
segmentation of the SC and traumatic axial MRI contusion in T2 in a cohort of patients 
with acute SC injury. They have developed a image analysis pipeline integrating 2D 
convolutionary neuron networks for the segmentation of the entire and intramedullary 
spinal cord lesions. 

In Article of [4] to segment the lesions in the pathology of multiple sclerosis, they 
studied two stages of pretreatment based on skull stripping (SS) and the improvement 
of the contrast which are two important steps to improve the quality rate of MS lesion 
segmentation. After the pretreatment step, a segmentation approach based on the method 
of maximizing expectations (EM) has been applied to extract SEP lesions. Again study 
of [5] for the contrast improvement of MRI images for a better appearance of normal 
tissues and sick tissues affected by the pathology of MS. 

The important recent segmentation study presents a new protocol to develop bench- 
mark segmentations of white matter tumors based on multi-rater consensus using a new 
MR database of 30 patients with multiple sclerosis [6]. They have developed advanced 
BrainSeg3Dsoftware which allows their correct and effective delineation in 3D MR 
images. 

From this study, we carried out work based on the tools of the BrainSeg3D software 
and this new database: the first work is to create a novel protocol applied the tools of the 
BrainSeg3D software applied on this new database which contains 30 MS patients. 

The new database is tested by the metaheuristic ant colony algorithm used by [7, 8] 
helping to improve the parameters of this algorithm [9, 10]. 

The research [10] manages to realize a new ACO_BrainSeg3D segmentation algo- 
rithm that combines the ACO algorithm and the protocol based on BrainSeg3D tools, 
this approach is tested on the novel longitudinal dataset with reference. 

Another recent study [11] objectively validate and compare several intensity-based 
methods for detecting white matter tumor changes using longitudinal MR image datasets 
with accurate ground truth. 

The Validation of change detection focused on evaluating the two main steps of 
change detection, namely Dissimilarity Map (DM) creation and segmentation. Therefore, 
combinations of three methods for DM creation (Subtraction of intensity images (STI), 
Generalized Like Lihood Ratio (GLR) and Logistic Regression Model (LRM)) and three 
for DM segmentation (Confidence level thresholding (CLT), Change vector angular 
histogram thresholding (CVAHT) and optimal threshold) were tested on 20 MR image 
datasets with a common pre-processing and post-processing. 

It was created by two specialist evaluators who subtracted and segmented of white 
matter tumor of two studies: baseline and follow-up Tl w, T2w and FLAIR MR Images. 
The reference segmentations were cooperatively treated by the evaluators till they 
heard on what was inspected the reference segmentation. This publicly dataset and 
the BrainSeg3D software are available on site web http://lit.fe.uni-lj.si/tools. 

In this study, the objective is to employ the proposed algorithm ACO_BrainSeg3D on 
anew longitudinal dataset containing 20 MS patients in order to evaluate the performance 
and reliability of this algorithm against the ground truth and other tested methods on the 
same database. 
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The aim of the actual work was to supply a literal test and to compare the results of 
propound algorithm with the three intensity based approaches Dissimilarity Map creation 
and segmentation combinations (STI/CLT, STI/CVAHT and STI/Optimal threshold), 
(GLR/CLT, GLR/CVAHT and GLR/Optimal threshold) and (LRM/CLT, LRM/CVAHT 
and LRM/Optimal threshold) [11]. 

We can thus note that almost all the outliers are identified by the ACO_BrainSeg3D 
protocol in front of the reference, and compared to three methods for DM creation (STI, 
GLR and LRM) combined with three for DM segmentation (CLT, CVAHT and optimal 
threshold) in terms of the median coefficient of similarity. 

We have tested the proposed algorithm using public longitudinal available dataset 
with brain MR Images containing two studies: baseline and Follow-up studies with 
three MRI sequences T1-w, T2-w and Flair. In all cases, our algorithm outperformed 
competing approaches. The obtained evaluations reported good results, where all the 
outliers have emerged in spite of there being falsely spotted outliers. 


3 The Proposed ACO BrainSeg3D Algorithm 


In this paper we expose the suggested method which used the combination of ACO 
algorithm with a propound protocol by using existing and new semi-automated outliers 
segmentation tools of BrainSeg3D software. So as to facilitate the tumors segmentation, 
we employed the MATLAB software and the specialized BrainSeg3D software. Our 
proposed contribution is named ACO_BrainSeg3D. 

The global function of the propound contribution is detailed on Fig. 1. 

The proposed algorithm is essentially based on two algorithms as detailed below: 

All MRI images from the database used are first processed by the ACO metaheuristic 
algorithm. Also in the studies [9, 12, 13] which are based on ACO algorithm with the 
same database, when we obtained good results compared to reference of the novel MR 
Datasets. 

Concerning this algorithm, our contribution is mainly focused on the variation of 
the parameters empirically in order to have good results for the segmentation of the 
images of the new longitudinal database and a good detection of MS lesions. The latter 
consists of three parameters to better characterize good quality segmentation. Then, the 
segmentation thresholds maximizing the criteria are sought using the ACO algorithm. 

The settings and the essential equations of ACO algorithm are detailed in our studies 
[9, 10]. Then the major tested values to do all simulations are presented in table 1. 

Our important contribution is the testing of parameters in order to obtain good results. 
This task is done in [12, 13] and it evaluated according to existing research and the 
consensus of segmentation. 


4 Validation Dataset 


The validated dataset contains two studies baseline and follow-up MR images for 20 
subjects with multiple sclerosis as illustrated in Fig. 2. 

The images were acquired on a 1.5 T Philips MRI machine at the University Medical 
Centre Ljubljana (UMCL). 
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Table 1. ACO algorithm setting. 


Values 
20*N 
Threshold number 


3 
Evaporation Coefficient p | 0.1 
COE 
B 1 [0 


ACO Algorithm 


Proposed 
Protocol based on 
BrainSeg3D tools 


Fig. 1. Workflow of ACO BrainSeg3D. 


(a) (b) (c) (d) 


Fig. 2. Longitudinal database (a) patient 1 to patient 6, (b) patient 7 to patient 12, (c) patient 13 
to patient 18 (d) patient 19 to patient 20. 
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Every subjects of this dataset contains a 2D T1-weighted, 2D T2-weighted and a 2D 
FLAIR image with different setting indicated in the Table 2. 

The reference of MR images was proposed by two expert evaluators for evaluation 
purposes. The 20 MS patients with baseline and follow-up studies with respectively 
T1-W, T2-W and Flair images are available in the site http://lit.fe.uni-lj.si/tools. 


Table 2. MRI image of database parameters. 


Repetition Time (TR) Spm — —] 4500ms 11,000ms 


Echo Time (TE) 
men. Pec ee 
Flip angle(FA) 


Sampling ————— 9x0.9x30mm a 45x0.45x3mm wa 9x0.9x3mm 


5 Results and Discussion 


In order to validate the capability of our algorithm to detect MS tumor, we calculated 
three essential terms of validation: the coefficient DSC, TPR (True positive Rate) and 
robustness of FPR (False Positive Rate). 

The True Positive Rate corresponds to sensitivity: 


True Positive Rate = TP/(TP + FN) (1) 


The False Positive Rate corresponds to precision that explains the capability to select 
all existing outliers corresponds to the value: 


False Positive Rate — TP/(TP 4- FP) (2) 


The Dice Similarity Coefficient also corresponds to the Kappa Index (or Dice coef- 
ficient) and measures the overlap between the test of segmentation obtained and the 
consensus. The Dice Similarity Coefficient (DSC) [15] can therefore be calculated with 
respect to consensus segmentation: 


DSC = (2 x TP)/((2 x TP) + FP + FN) (3) 


Our important goal is to detect all outliers, True positive (TP) presents a detected 
region if at least one voxel overlaps the reference, otherwise it is considered a false 
positive (FP). Any labeling not spotted by our approach is considered as a false negative 
(FN) [16]. Whose values fluctuate in [0, 1] and a value superior 0.6 presented a good 
conformity with the ground truth. 

We see here in Fig. 5 that the proposed algorithm manages to detect almost all the 
lesions in the axial slices of the original image as for patients 1, 3, 6, 8 and 11. For patients 
7 and 19, there are slightly visible lesions not detected. And for the other patients, they 
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Fig. 3. Minimum and maximum values of DSC, FPR and TPR of longitudinal database. 


are considered as healthy patients due to the absence of lesions; we can only discover 
this for a single patient 17 in which the proposed algorithm detects a small, slightly 
visible lesion. 

The results of segmentation by the suggested algorithmACO_BrainSeg3D are 
exposed in Fig. 5. This algorithm yields good results, where all the outliers have arisen 
in spite of there being falsely spotted lesions. 

The respective values procured with our proposed ACO_BrainSeg3D algorithm of 
DSC are between 0.44 and 1, the values of FPR are in the interval [0.5 1] and the values of 
TPR are between 0.33 and 1(Fig. 3). We mark that the validation values obtained in this 
test are lightly elevated than these reported on every outliers segmentation publicized in 
the literature [11]. 

We managed in this paper to detect several lesions according to the number and 
volume in the first phase of advancement of the lesions it is the relapsing remittent form 
RR for all the patients of the database with a percentage of 70% of rate of advancement, 
as we detected lesions in the secondary progressive form SP with a portion of 5% and 
a percentage of 25% of other patients not specified or we cannot identify whether these 
patients are really healthy or there are very small lesions in the early phase that we 
couldn’t detect (Fig. 4). 

This analysis has been processed for some patients in the database for example for 
the axial sections of the flair image for patients 2, 4, 5, 20 ... as illustrated in Fig. 5. 

This is due to the fact that we can see the positions of further outliers in link with 
the consensus, and we can spot the missed outliers, which could otherwise have a strong 
impact on DSC. 

We objectively evaluated in this paper the proposed algorithm which combine ACO 
algorithm with the protocol using the BrainSeg3D tools on longitudinal MRI database 
of 20 MS subjects. 
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The aim of the actual work was to supply a literal test and to compare the results of 
propound algorithm with the three intensity based approaches Dissimilarity Map creation 
and segmentation combinations (STI/CLT, STI/CVAHT and STI/Optimal threshold), 
(GLR/CLT, GLR/CVAHT and GLR/Optimal threshold) and (LRM/CLT, LRM/CVAHT 
and LRM/Optimal threshold). 


40 +4 M MS cases 


30 + M rate of progress 96 


o | nad 


RR SP not specified 


Fig. 4. Advancement rate of 20 patients in the database. 


To explain the impact on the median DSC, Table 3 reports corresponding median 
values of the proposed algorithm and the combinations methods based on segmentation 
and creation DM after the post-processing. 

The median DSC after post-processing of three combinations of three change 
segmentations DM and three creation DM approaches was consistently reduced in 
accordance of our suggested algorithm where the value DSC is 0.76. 

The middle DSC value for the ACO BrainSeg3D algorithm is studied as follows: 


20 
DSCmoy = 1/20 PM DSCi (4) 


Aside from various fusions of the DM creation and segmentation approaches were 
tested by using the same quantitative metrics. Comparative test according to three DM 
alternatives (Table 3) illustrated an elevated average DSC (76%). 

This result reports that the used outliers segmentation algorithm marked very 
substantial and reliable tumor segmentations. 

That our segmentation algorithm of segmentation applied on the novel longitudinal 
database is significantly higher compared to the DSC of DM creation combined with three 
DM segmentation. So the ACO BrainSeg3D algorithm detects more lesions compared 
to ground truth. 
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(a) (b) (c) 


Fig. 5. Axial sections of corresponding FLAIR images of 20 patients (1** column), the results of 
our proposed Algorithm (24 column) and the ground truth segmentation (3d column) for each 
block. (a): patients 1 to 7; (b): patients 8 to 14 and (c): patients 15 to 20. 


Table3. Values of median DSC 


Method STI/CLT STI/CVAHT STI/Optimal Proposed 
threshold Method 

Median DSC 0.52 0.19 0.58 0.76 

metric 

Method GLR/CLT GLR/CVAHT GLR/Optimal Proposed 
threshold Method 

Median DSC 0.51 0.49 0.57 0.76 

metric 

Method LRM/CLT LRM/CVAHT LRM/Optimal Proposed 
threshold Method 

Median DSC 0.38 0.36 0.45 0.76 


metric 
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The advantage of proposed algorithm ACO_BrainSeg3D applied on this longitudinal 
database is the detection of almost all existing lesions in the ground truth thanks to the 
values carried out within the acceptable measurement interval. Our algorithm needs more 
run time but we don’t have a real time constraint. The only drawback is the non-detection 
of small lesions which are not very visible, which encourages the researchers to improve 
their techniques until they detect small lesions of a small volume. 

We can thus note that almost all the outliers are identified by the ACO_BrainSeg3D 
protocol in front of the reference, and compared to three methods for DM creation (STI, 
GLR and LRM) combined with three for DM segmentation (CLT, CVAHT and optimal 
threshold) in terms of the median coefficient of similarity. 

This validation is carried out first based on the DSC coefficient largely which indi- 
cates similarity and second concerning the ability and precision to detect the number of 
tumors existing in the ground truth based on the TPR and the FPR. 

In order to mark this object and allow other researchers to replicate the test of this 
work, we wish to extend and publicly propagate the encouraging results of our proposed 
algorithm using the longitudinal database available on the site: http://lit.fe.uni-lj.si/tools. 

We consider this study as the first but important step towards the realization of a 
protocol based on BrainSeg3D software tools combined with an improved metaheuristic 
algorithm applied to different public databases for validation and comparison of detection 
methods with longitudinal and other changes. 


6 Conclusion 


In this paper, we propounded an automatic protocol for cerebral tumors segmentation in 
longitudinal MRI images. The idea was evaluated the ACO algorithm by using the MAT- 
LAB software and protocol based on semi-automated tools using BrainSeg3Dsoftware 
with a for longitudinal MRI image dataset which contains 20 patients with multiple 
sclerosis with reference agreed by two domain specialist. 

We have tested the proposed algorithm using public longitudinal available dataset 
with brain MR Images containing two studies: baseline and Follow-up studies with 
three MRI sequences T1-w, T2-w and Flair. In all cases, our algorithm outperformed 
competing approaches. The obtained evaluations reported good results, where all the 
outliers have emerged in spite of there being falsely spotted outliers. 

A short-term perspective is to make an interface that implements our proposed algo- 
rithm ACO_Brainseg3D on computer. This can facilitate the use of the method and gain 
more time. It is necessary to test this interface on larger databases to clear the strengths 
and limitations. This step is essential before exporting our algorithm in a clinical context. 
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Abstract. As the number of people with disabilities continues to increase, diffi- 
culties in dressing activities for people with disabilities are also increasing. How- 
ever, the currently available clothing for the disabled do not satisfy their functional 
and aesthetic needs. Therefore, this study aims to contribute to vitalizing the devel- 
opment of clothing for the disabled in Korea by collecting domestic study on the 
development of clothing for the disabled and analyzing them in various areas. 
Finally, 5 studies were selected. As a result of classifying into 4 areas of body 
type, considerations, points for improvement, and design, it can be used as basic 
data for study related to the development of clothing for the disabled. 


Keywords: People with disability - Wheelchair users - Clothes - Development - 
Design 


1 Introduction 


According to the Ministry of Health and Welfare data, as of the end of 2021, the number 
of registered disabled people was 2.645,000 (5.196 of the total population), an increase 
of 12,000 compared to the end of last year. As the number of people with disabilities 
continues to increase in Korea, the difficulty of wearing clothes for the disabled is also 
increasing [1]. Clothing for the disabled is clothing that helps disabled people to wear 
clothes conveniently [2]. For example, it should be able to move more freely when 
pushing a wheelchair, and comfort should be ensured even when sitting for long periods 
of time while wearing clothing [3]. In addition, it is necessary to develop clothes that 
allow people with disabilities to wear clothes with a minimum of help from others, 
and to minimize the inconvenience of putting on and taking off themselves, so that the 
inconvenience of clothes felt by people with disabilities should be resolved. However, 
the currently available clothes for the disabled do not satisfy their functional needs as 
well as their aesthetic satisfaction [4]. And compared to developed countries, research 
and development related to clothing for the disabled in Korea is very insufficient, and 
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the distribution of clothing considering the inconveniences of the disabled is also very 
limited [5]. 

Disabled people who use wheelchairs spend most of their time in a seated position in a 
wheelchair as they become dependent on a wheelchair due to lower extremity amputation 
or paraplegia due to congenital or acquired diseases and accidents [6]. Because of this, 
physical changes appear, and it is difficult for the disabled to purchase clothes suitable 
for their body type, so there is a limitation in that the types of clothes that are purchased 
and worn are items that are convenient to put on and take off [7]. In addition, people with 
disabilities in wheelchairs want to increase their opportunities for activities and social 
participation by wearing clothes that are indistinguishable from non-disabled people in 
appearance because their body image is distorted or their self-esteem is relatively low 
[8]. 

Therefore, this study aims to contribute to vitalizing the development of clothing for 
the disabled in Korea by collecting domestic study on the development of clothing for 
the disabled and analyzing them in various areas. 


2 Method 


2.1 Collecting Data 


This study was conducted as a literature search method. In order to investigate studies 
related to the development of clothing for the disabled in Korea, the DBpia, KISS, 
RISS, and KCI databases were used to collect documents published in domestic journals. 
Keywords used in the search included search terms such as ‘disabled person’, ‘wheelchair 
user’, ‘clothes’, ‘development’, and ‘design’. In addition, missing articles were included 
by re-searching the references of the collected articles. 


Search Key Term : "disabled person OR wheelchair user AND clothes AND development OR design " 


Records identified through databases searching (n = 36): 
DBpia (n = 4), KISS (n = 11), RISS (n = 14), KCI (n = 7) 


| 


Records after duplicates removed 
(n=16) 


| 


Records screened (n = 16) —Ó Records excluded (n - 4) 
* No content about key word in the title (n = 4) 


Full-text articles assessed for eligibility * Not related to clothing development (n = 1) 
(n = 12) * Clothing development for special people(n = 1) 


* Not an experiment research (n = 5) 


Studies finally included in analysis (n = 5) 


Fig. 1. Study process 
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2.2 Data Analysis 


A total of 36 researches were published from 2012 to December 2021, and DBpia(4), 
KISS(11), RISS(14), and KCI(7) were searched for collected through literature search. 
After that, five studies were finally selected through a study process (Fig. 1). Analysis 
standards and methods were configured to be suitable for consideration on research 
on the development of clothes for the disabled in Korea by identifying the contents of 
the collected studies. The research subjects were divided into four areas: body shape, 
considerations, points for improvement, and design. In order to understand the contents 
of each item, each research study was read and detailed contents were grasped. 


3 Results 


3.1 Year of Publication 


There are a total of 5 studies on research related to the development of clothes for the 
disabled in Korea published from January 2012 to December 2022, 3 in 2013 and 2 in 
2014. 


3.2 Subjects 


The age of subjects related to the development of clothes for the disabled in Korea 
was 40—49 years old the most at 4396, followed by 30—39 years old, 50—59 years old, 
20—29 years old, and 60 years old or older. 

Looking at the types of disabilities of the subjects, spinal cord injury accounted for 
the most at 48%, followed by amputation, polio, and brain lesion disorders. 

The subjects’ use of assistive devices was frequent use of wheelchairs the most at 
44%, followed by walking aids, crutches, no use, and canes. 


3.3 Clothing Body Type 


The body type of the wheelchair disabled person's clothing is largely divided into 4 
in the upper extremity and 2 in the lower extremity. The upper extremity body type is 
divided into a normal body type (A), a thick waist type (B), a large belly type (BB), 
and an inverted triangle (Y). And the lower extremity body type is divided into a normal 
body type (A), a thick waist type (B). The distribution according to the subject's clothing 
body type is shown in Figs. 2 and 3. Looking at the clothing body type, the distribution 
of upper extremities was the most common a normal body type (A) with 34%, followed 
by a body with a large belly (BB), a body with a thick waist (B), and an inverted triangle 
(Y). And the distribution of the lower extremity body type was 70% for those with a 
thick waist (B), more than those for the normal body type (A). 

This result can be explained by the fact that the upper extremity exercise increases 
because the upper extremities must be continuously pushed while moving the wheelchair, 
and the lower extremity exercise decreases by relying on the wheelchair. 
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Fig. 2. Upper extremity body type 
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Fig. 3. Lower extremity body type 


3.4 Considerations When Purchasing Clothing 


Considerations when purchasing clothes are divided into aspects of functionality and 
practicality and aesthetic. Table 1 shows the distribution of considerations when purchas- 
ing clothes for subjects. First, in terms of functionality and practicality, it appears that 
‘very much considered’ considers the convenience of putting on and taking off, which is 
the most distributed. Next, ‘very much considered’ and ‘much considered’ were widely 
distributed in the order of dimensional conformity, usability of clothes, and ease of repair. 
It can be seen that the low price is not a major consideration when purchasing clothes, 
as it is widely distributed in ‘much considered’ and ‘so-so considered’. 

In terms of esthetic, it is shown that ‘very much considered’ considers the coverage 
of the disability area, which is the most distributed. Next was the palatability in which 
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a ‘very much consideration’ and ‘much consideration’ were widely distributed. Fashion 
trends are widely distributed in ‘so-so considered’, so it can be seen that this is not a 
major consideration when purchasing clothes. 


Table 1. Considerations when buying clothing 


Division N (96) 
Functionality and practicality | Convenience of putting on | very much considered | 46 (45) 
and taking off much considered 4] (40) 
So-so 16 (15) 
Dimensional conformity very much considered | 35 (35) 
much considered 45 (45) 
So-so 20 (20) 
Ease of repair very much considered |22 (24) 
much considered 46 (50) 
So-so 24 (26) 
Clothing usability very much considered |21 (22) 
much considered 57 (60) 
So-so 17 (18) 
Low price very much considered | 11 (12) 
much considered 45 (49) 
So-so 36 (39) 
Aesthetic Palatability very much considered |25 (26) 
much considered 38 (40) 
So-so 32 (34) 
Disability area coverage very much considered | 27 (29) 
much considered 31 (33) 
So-so 36 (38) 
Fashion trends very much considered | 12 (15) 
much considered 21 (25) 
So-so 49 (60) 


3.5 Clothing Improvements 


The points for improvement of clothes were divided discomforts and repair parts for 
clothes. Among them, the discomforts were divided into upper and lower discomforts, as 
shown in Table 2. First, looking at the discomforts of upper, the most common discomfort 
about jacket length was 31%, followed by sleeve length, extra space, upper length, and 
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sleeve circumference. Looking at the discomforts about the lower, the most common 
discomfort about the length of the pants was 27%, followed by the rise length, waist 
circumference, hip circumference, and knee circumference. 

The repair part was divided into the upper and the lower part similarly to the dis- 
comfort, and it is shown in Table 3. When looking at the repair part of the top, 38% of 
the respondents said that the sleeve length needs to be repaired, followed by fastening, 
jacket length, and armpit slack. When looking at the repair part of the bottom, the opin- 
ion that the trouser length needs to be repaired was the most at 48%, followed by the 
circumference of the lower, the length of the rise, the closure, and the circumference of 
the hip. 

This result can be explained by the fact that if the jacket length is too long, the jacket 
may get caught on the wheelchair wheels when the wheelchair is pushed. And if the 
pants length is too long, this can be explained by the fact that the pants may drag on the 
floor when moving in a wheelchair. 


Table 2. Upper and lower discomfort 


Division N (96) 

Upper Sleeve circumference 46 (11) 
Extra space 87 (20) 
upper length 49 (11) 
Sleeve length 115 (27) 
Jacket length 133 (31) 

Lower Waist circumference 159 (21) 
Rise length 176 (23) 
Hip circumference 114 (15) 
Pants length 206 (27) 
Knee circumference 104 (14) 


3.6 Clothing Design 


Clothing design was largely divided into color, adjustment location, fastening, and fin- 
ishing, as shown in Table 4. Looking at the color design, most preferred achromatic 
colors at 58%, followed by colored colors and pastel colors. Looking at the adjustment 
location design, most preferred the front adjustment at 74%, followed by no adjustment, 
shoulder adjustment, and armpit adjustment. Looking at the design of the fastening, the 
button was the most preferred at 5196, followed by the zipper and Velcro. Looking at 
the finished design, 37% of the respondents preferred to do nothing like general clothes, 
followed by buttons, rubber bands, and Velcro. 

In the case of color design and finishing design, it can be explained that the disabled 
people want to wear clothes that are indistinguishable from non-disabled people in 
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Table 3. Repair part of upper and lower 


Division N (%) 

Upper Fastening 39 (22) 
Sleeve length 66 (38) 
Extra space 19 (11) 
Armpit 22 (13) 
Jacket length 29 (16) 

Lower Fastening 31 (13) 
Rise length 35 (15) 
Hip circumference 20 (8) 
Pants length 113 (48) 
Pants circumference 38 (16) 


appearance. And it can be explained that the reason for preferring the front adjustment 
is that it is convenient when going to the bathroom because the study subjects are all 
male. 


Table 4. Clothing design 


Division N (906) 
Color Pastel color 28 (9) 
Achromatic color 183 (58) 
Colorful 104 (33) 
Adjustment location No adjustment 54 (17) 
Front adjustment 231 (74) 
Adjustment below armpit 16 (5) 
Shoulder adjustment 13 (4) 
Fastening Button 334 (51) 
Zipper 245 (37) 
Velcro 78 (12) 
Finishing Common 174 (37) 
Button 125 (26) 
Rubber band 94 (20) 
Velcro 82 (17) 
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4 Conclusion 


Through the results of the study, it was confirmed that the development of clothes for the 
disabled in Korea was carried out in terms of body type, considerations, improvement 
points, and design. However, there are limitations in that the subjects are all male and 
the number of studies is small. Therefore, it is necessary to collect studies with a wider 
period for follow-up studies, and the gender of the subjects should be included in a 
balanced way. And if additional research on clothing for people with disability, such 
as additional research on body type according to the type of disability, is conducted 
in the follow-up study, more systematic research data can be established. This study is 
meaningful in that it was able to analyze and grasp studies related to the development 
of clothing for the disabled, and it will be used as basic data for related research. In the 
future, research related to the development of clothes for the disabled will be conducted 
in various fields, and it is expected that the results will increase the convenience of the 
disabled and activate the social participation of the disabled. 


Acknowledgments. The research was supported by a grant of the Korea Health Technology R&D 
Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry 
of Health & Welfare, Republic of Korea (grant number: HJ21C0014). 


References 


1. Esmail, A., et al.: The role of clothing on participation of persons with a physical disability: a 
scoping review. Appl. Ergon. 85, 103058 (2020) 

2. Azher, N., Saeed, M., Kalsoom, S.: Adaptive clothing for females with arthritis impairment. J. 
Univ. Med. Dent. Coll. 3(2), 52-59 (2013) 

3. Ikenyei, N.S., Amaechi, E.N.: Persons with disability and the 2030 agenda of sustainable 
development goals: the need for organized intervention through adaptive clothing to synthesise 
elaborate social inclusion in Nigeria. Int. J. and Soc. Inq. 6(1) (2020) 

4. Soon-i Bae: A preference study for the development of clothing for men with lower extremity 
amputation. Fashion Knit 11(1), 81-89 (2013) 

5. Jin, K.: A study on satisfaction and improvement points for ready-made clothing for the devel- 
opment of men's wear for the physically disabled: focusing on the amputees of the lower 
extremity. Des. Knowl. J. 27, 189-201 (2013) 

6. Park, K., Park, J., Yang, C., You, H.: A study of body shapes to design clothing for wheelchair 
users. In: Proc Fall Conf the Ergonomics Society of Korea, pp. 263-270 (2013) 

7. Lee, J.-H., Yang, J.-E., Park, K.-A., Park, J.-W., Yoo, H.-C., Bae, B.-J.: Analysis of the wearing 
conditions and preferences of men in wheelchairs for the development of suits. J. Korean Soc. 
Cloth. Text. 38(5), 733-742 (2014) 

8. Yang, J.-e., Lee, J.-h., Park, K.-a.: A study on domestic and foreign apparel products, thesis, 
and patents for improving the functionality of wheelchair disabled people's clothing. Korea 
Design Forum 42, 7-16 (2014) 


254 H. Yang et al. 


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 
International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, 
adaptation, distribution and reproduction in any medium or format, as long as you give appropriate 
credit to the original author(s) and the source, provide a link to the Creative Commons license and 
indicate if changes were made. 

The images or other third party material in this chapter are included in the chapter’s Creative 
Commons license, unless indicated otherwise in a credit line to the material. If material is not 
included in the chapter’s Creative Commons license and your intended use is not permitted by 
statutory regulation or exceeds the permitted use, you will need to obtain permission directly from 
the copyright holder. 


Short Contributions: Wellbeing 
Technology 


® 


Check for 
updates 


Empowering Well-Being Through 
Conversational Coaching for Active and Healthy 
Ageing 


Michael McTear! ©), Kristiina J okinen?, Mohnish Dubey?, Gérard Chollet?, 
Jéróme Boudy*, Christophe Lohr’, Sonja Dana Roelen?, Wanja Móssing?, 
and Rainer Wieching? 


! Ulster University, Belfast, UK 
mf.mctear@ulster.ac.uk 
2 Artificial Intelligence Research Center, (AIRC/AIST), National Institute of Advanced 
Industrial Science and Technology, Tokyo, Japan 
kristiina.jokinen@aist.go.jp 
3 Institute fiir Angewandte Informatik, Germany (INFAT), Leipzig, Germany 
dubey@infai.org 
4 Institut Mines-Télécom (IMT), Palaiseau, France 
{gerard.chollet, jerome.boudy}@telecom-sudparis.eu 
5 Institut fiir Experimentelle Psychophysiologie Gmbh (IXP), Diisseldorf, Germany 
{s.roelen,w.moessing}@ixp-duesseldorf.de 
6 Universität Siegen (USI), Siegen, Germany 
rainer.wieching@uni-siegen.de 
7 Institut Mines-Télécom (IMT), Brest, France 
christophe.lohr@imt-atlantique.fr 


Abstract. With life expectancy growing rapidly over the past century, societies 
are being increasingly faced with a need to find smart living solutions for elderly 
care and active ageing. The e-VITA project, which is a joint European (H2020) 
and Japanese (MIC) funded project, is based on an innovative approach to vir- 
tual coaching that addresses the crucial domains of active and healthy ageing. 
In this paper we describe the role of spoken dialogue technology in the project. 
Requirements for the virtual coach were elicited through a process of participa- 
tory design in workshops, focus groups, and living labs, and a number of use cases 
were identified for development using the open-source RASA framework. Knowl- 
edge Graphs are used as a shared representation within the system, enabling an 
integration of multimodal data, context, and domain knowledge. 


Keywords: Active and healthy ageing - Dialogue system - Knowledge graphs - 
Participatory design 


1 Introduction 


With life expectancy growing rapidly over the past century, societies are being increas- 
ingly faced with new challenges related to an ageing population and the need to find 
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smart living solutions for elderly care and active ageing [1]. This paper presents an 
innovative approach involving virtual coaching that is currently being developed in a 
3 year joint European (H2020) and Japanese (MIC) funded research project!:?. The 
virtual coach addresses the crucial domains of active and healthy ageing in cognition, 
physical activity, mobility, mood, social interaction, leisure, and spirituality, with the 
aim of empowering older adults to better manage their own health and daily activities, 
resulting in improved well-being and improved stakeholder collaboration. The virtual 
coach provides individualized profiling and personalized recommendations based on big 
data analytics and social-emotional computing, detecting risks in the user’s daily liv- 
ing environment by collecting data from external sources and non-intrusive sensors and 
providing support through natural interactions with 3D-holograms, emotional objects, 
or robotic technologies using multimodal and spoken dialogue technology, advanced 
knowledge graph representations, and data fusion. [2] provides a general overview of 
the project. 

The remainder of the paper is structured as follows. Section 2 reviews related work in 
the area of active and healthy ageing, with particular reference to the use of dialogue sys- 
tems to provide a virtual coaching application. Section 3 presents the technologies being 
employed in the e-VITA virtual coach, looking at multimodal data fusion from sensors, 
emotion detection, knowledge graphs, and dialogue technology. Section 4 describes the 
development of an initial prototype system, including a process of participatory design 
resulting in use cases and content development, some of which has been implemented 
in the Wave 1 prototype. Section 5 concludes by outlining the next steps in the project. 


2 Related Work 


There has been increasing interest in recent years in exploring how new developments 
in conversational AI and socially assistive robots can be applied to support active and 
healthy ageing. [3] presents a comprehensive state-of-the-art review in which several 
future research directions are identified, including: the need for unconstrained natural 
language processing and conversational strategies to enable robust and meaningful two- 
way conversations; the ability to interpret affective modalities in order to enhance user 
engagement and trust; and challenges in deployment of ensuring user adherence and data 
privacy. Looking at some individual relevant projects, [4] reports a month-long study 
of a virtual agent and robot that could interact with older adults in their homes through 
dialogue and gestures with the aim of providing companionship and reducing isolation. 
The importance of dialogue capability to enable social robot agents to provide natural 
interaction is also emphasized in [5], while [6] describes how information about the older 
adult's emotional status was extracted from an analysis of their verbal and non-verbal 
communication. [7] is an example of an ongoing project in the Netherlands with the aim 
of improving the lives of older adults through the use of voice technology. The e-VITA 
project also focusses on the issues addressed in these examples of related work but in 


l https://www.e-vita.coach/. 

2 The e-VITA project has received funding from the European Union H2020 Programme under 
grant agreement no. 101016453. The Japanese consortium received funding from the Japanese 
Ministry of Internal Affairs and Communication (MIC), Grant no. JPJ000595. 
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addition extends the research to examine how data from sensors and emotion analysis 
can be used within dialogues with the older users, and how knowledge graphs can be 
used to store data relevant to active healthy ageing as well as personal data about the 
user to enable the virtual coach to offer personalized information and recommendations. 


3 Technologies and Architecture of the Virtual Coach 


The complete e-VITA platform is based on the Digital Enabler platform provided by 
the project partner Engineering (Italy). The platform is designed to include multiple 
devices and software components from across Europe and Japan that are based on differ- 
ent technologies and standards. The platform supports communication and integration 
among different smart devices such as sensors and robots, as well as the collection and 
management of data to provide coaching functionalities. 


3.1 Multimodal Data Fusion from Sensors 


In a smart home, sensors can be used to capture and monitor data in order to provide 
the user with assisted, safe and comfortable living. In the e- VITA project three types of 
sensors are used: holter-based sensors worn by the user (or possibly his/her smartphone) 
that sense physiological and actimetric parameters; environmental sensors that measure 
physical data to assess the level of comfort and the quality of the indoor environment; and 
home-based sensors to monitor user behaviour and activities. Data from these sensors is 
combined (or fused) to make inferences about users in their environment, for instance, 
postures, localisation in the home, and the users' physiological states. Contextual infor- 
mation extracted from these sensors can be stored in knowledge graphs and exploited 
by the interactive voice-based coaching system. 

Various challenges being addressed include interoperability between the sensors 
involving a combination of multiple different data that may operate at different sampling 
frequencies and with different representation formats or scales for the targeted e- VITA 
applications, e.g., physical exercises, activities of daily living (ADL), or fall prevention 
[8]. Another goal is to perform Data Minimization for transmission of information into 
the Cloud. 

To date software modules have been implemented that deliver data fusion to assess 
the user's situation (location, activity, vital state) and environmental conditions (temper- 
ature, humidity). The delivered data fusion software uses the Digital Enabler framework, 
aiming to provide semantical situation and environment labels that can be used by the 
knowledge graphs and the dialogue system. The Data Fusion algorithms constitute a first 
step for processing heterogeneous sensor data and signals in a multimodal way. Future 
work will address other modalities in order to obtain a complete set of data about the 
user's situation and environment. 


3 https://www.eng.it/en/our-platforms-solutions/digital-enabler. 
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3.2 Emotion Detection System 


A range of emotional cues can be detected in acoustic parameters in the user's voice, 
including parameters in the Frequency domain (e.g., fundamental frequency, jitter, or 
pitch), in the Time domain (e.g., speech rate, speech pauses and syllable rates), in the 
Amplitude domain (e.g., intensity or energy), and in the Spectral Energy domain (e.g., 
relative energy in different frequency bands) [9]. The Emotion Detection System (EDS) 
that is employed in the e-VITA platform enables the detection and classification of 
frequently used basic emotions such as fear, anger, disgust, happiness, surprise, and sad- 
ness during interactions between the coaching system and older adults [10]. The detected 
emotions can then be used by the dialogue system to provide appropriate interventions 
[11]. 

The exact classes detected by the e- VITA EDS may vary between the target languages 
in the e- VITA consortium depending on the availability of training data. In the current 
release, the German variant of the EDS can detect anger, disgust, fear, joy, neutral, and 
sadness. The Japanese variant can detect anger, contempt, disgust, fear, joy, neutral, 
sadness, surprise, and trust. 

Within the e-VITA project, the EDS may need to be executed in a range of envi- 
ronments depending on upload bandwidth, data minimization, privacy concerns, and 
prototype environments. Thus the EDS may be implemented as a component within the 
Digital Enabler cloud environment, but could alternatively also be implemented on a 
local edge-computing device with yet-to-be-defined hard-/software specifications. 


3.3 Knowledge Graphs 


Knowledge Graphs are used to represent the knowledge that is used by intelligent AI 
systems [12]. A Knowledge Graph (KG) stores data about real-world things, events, 
and concepts. For example, an entity in a KG contains a semantic description and many 
characteristics that relate it to other entities, concepts, or events. Entities in a KG are 
represented as nodes and the properties of the entities are represented as edges in the 
graph that relate the nodes, resulting in a network of enhanced knowledge for a specific 
genre or topic. 

In the current project the KG stores data about the user and the environment that is 
relevant to active healthy ageing and that enables the virtual coach to offer personalized 
information and recommendations. A localized KG stores personal information and is 
connected to the central KG. The central KG stores information required for various 
functions that are be used across the platform. The central KG is connected to a database 
that stores procedural details that cannot be harnessed through KG triples. The local KG 
is on-premise and keeps the personal data safe, while being able to offer assistance and 
a personalized user experience to an individual user. For instance, the fact that a user is 
male is stored in the local KG enabling the entire system to recommend activities suitable 
for male users. Figure 1 presents an excerpt of the graph that describes the prevention 
domain and the medical examinations that a male or female can take. 
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Fig. 1. Excerpt from the knowledge graph for the medical prevention domain 


3.4 Dialogue System 


The Dialogue System is the key component that enables interactions with the user. Its 
functions include understanding the user's input messages and providing an appropriate 
responses in the given context (including the previous dialogue context and environ- 
mental state) [13]. In the e- VITA project the Dialogue Manager is implemented using 
the RASA Open Source Framework that will be integrated with speech technology 
components to enable spoken interaction. RASA supports a machine learning-based 
Natural Language Understanding pipeline for the interpretation of the user's inputs and 
a combination of rule-based and machine learning-based dialogue policies to deter- 
mine the system's actions [14]. Knowledge Graph technology is integrated via specific 
knowledge-based actions supported by RASA to supply content for the system responses 
by querying the domain knowledge base [15]. 


4 Prototype Virtual Coach 


A prototype version of the virtual coach was delivered at month 15 of the project. This 
section outlines how the requirements for the prototype were gathered and analysed, 
followed by a discussion of the use cases addressed in the Wave 1 prototype and an 
example of an interaction with the virtual coach. 


4.1 Requirements Gathering and Analysis 


Requirements for the virtual coach were gathered in interviews with older community- 
dwelling adults in Germany, Italy, and Japan, all of whom described themselves as regular 
users of smartphones and personal computers. In the first phase users were invited to 
living labs, while in the second phase studies were conducted in the homes of users. 
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One of the studies conducted in Germany involved interactions with a nutrition chatbot 
and with the Nao robot in which a range of scenarios was explored including reminder 
functions, news, stories, and jokes, and general companionship. The end-user studies 
in Japan included interactions with real devices in Living Labs. One of these studies 
investigated users interacting with the Nao robot and a RASA dialogue system that 
included a general conversation of 10 min with the coach about daily living and a further 
10 min conversation about food and general question-answering about the news. The 
users were instructed to engage in several scenarios, for example: 


e Tell that you feel sad — the user expresses sad emotions and the coach aims to provide 
empathic and consoling companionship. 

e Tell that you want some exercise — the user tells the coach that he/she wants some 
exercise. The coach records the user's exercise preferences and provides options that 
can help improve the user's health and physical condition. 


Results from the studies included comments that the users would like to be able to 
engage in longer conversations and not just receive short answers to their questions. In 
some cases, the system received user responses that were not included in the original 
design and that would have to be included in the machine-learning models of the next 
version of the system. Also required was the implementation of information about the 
user's daily routines, events in the user's calendar, and user preferences about topics 
such as music, and reminders. 


4.2 Interactions with the Prototype Dialogue System 


Based on the interviews with users the following use cases were identified for the 
prototype system to be developed and evaluated in Wave 1 of the project: 


Daily support; 

Health activity support; 

Environmental monitoring support; 
Question-Answering over Wikipedia, News; 
Social Activity support. 


The following interaction is based on the RASA story represented in Table 1 which is 
activated following the recognition of the user's utterance “what preventive examination 
should I have performed by my medical doctor" as the intent ask examination. The 
system consults the knowledge graph shown in Fig. 1 and as ithe system does not know 
the user's gender in this use case, it asks the user to which the user replies “male”. 
The system follows the has right-to link from the male node in the knowledge graph, 
retrieves the required information, and outputs it to the user. A screenshot of the dialogue 
produced by this story is shown in Fig. 2. 
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Table 1. RASA story: preventive examinations. 


- story: preventive examinations, the male user would like to know what preventive measures 
he can receive 


steps: 


= intent: ask_examinations 


- slot_was_set: 


7 user_gender: null 


- action: utter_ask_gender 


- intent: say_gender 


= action: action_examination_KG #KG data 


- action: utter_suggest_examination 


Conversation session started on 24 Mar 2022 


action listen 


What preventive examinations should | have 


performed by my medical doctor? 


ask examinations("examination, type":"preventive examinations"intervention person":"medical doctor") (1.0) 
slot(*examination type":"preventive examinations") 


vtter ask gender (0.98) 


can you tell me if you are a male or female 
o please 
action listen (1.0) 
slot("user gender":"male*] 
o 
say gender(*user gender":"male"] (0.99) 


utter suggest examination (0.98) 


let me guide you. You have the right to a 
prostate examination and screening of the 


o abdominal aorta 


action listen (1.0) 


e e 


Start typing a message... KJ A g 


Fig. 2. Screenshot of dialogue in the medical prevention domain 


5 Next Steps 


In the next phase of the project the dialogue system will be tested and evaluated in the 
living labs and in the homes of the older adults. For the home-based trials the devices 
will be prepared by the study centres and installed in the homes of the users. A diary and 
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end-user manuals will be provided and the system will be explained and demonstrated. 
A human coach from the community will also visit once a week during the trials which 
will last for one month. 

Future technical work with involve more complex knowledge modelling and knowl- 
edge graph creation and integration of the dialogue system with new knowledge graphs, 
sensor data, and data from the emotion detection system. 
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Abstract. Smart Home Technology (SHT) as assistive technology (AT) is becom- 
ing an important active research field in the field of rehabilitation. For this purpose, 
Home Modification (HM) is one of the most common ways to improve the quality 
of life of the Persons with physical disabilities (PwPD). In this context, we propose 
anew Smart Home-based Home Modification Program (SHbHM) to improve the 
quality of life for PvPD. Our method simply uses Bluetooth or extends to Wi-Fi 
and Zigbee networks. A pilot study was conducted with five PwPD at home to 
investigate the effectiveness of the program. The reported results show a high qual- 
ity of life, and the occupational performance and satisfaction are greatly improved, 
indicating that it is an efficient alternative. 


Keywords: Smart home - Home modification - Quality of life - Disabled persons 


1 Introduction 


Persons with physical disabilities (PwPD) often have difficulty managing and access- 
ing their homes, and they must ask their caregiver for help to complete most of their 
life. Traditional home modification designed for PwPD such as Environmental Control 
Systems (ECS) can increase PwPD’s independence and quality of life (QOL). However, 
several barriers limited the use and effectiveness of ECS including complex usage, their 
high-cost, and lack of institutional arrangements and recognition that it is not necessary 
[1]. smart home technology is rapidly evolving, resulting in dynamic and affordable 
devices. In their study, smart home played a major role in improving PwPD's quality of 
life, occupational performance, and home management [2]. However, in Korea, there are 
only studies composed of low technology for fall prevention purposes or applied to test 
beds, but clinical studies applying smart home to homes are insufficient [4—7]. This pilot 
study aims to gather information on applicability, effectiveness, and limitations to pro- 
vide smart home-based home modification (SHbHM) to community practice areas as a 
first step in developing programs that promote quality of life, occupational performance, 
and home management. 


© The Author(s) 2022 
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2 Method 


This study was applied the SHbHM program to PwPD using one group pretest-posttest 
design and investigated the applicability, effectiveness, and limitations of the SHbHM 
program. 


2.1 Participants 


This study was selected by applying the objective sampling method. Through deliberate 
sampling, the study participants’ intentional selection can obtain a lot of information 
about the selection of topics and phenomena suitable for the study topic [8]. The selection 
criteria are as follows. First, those who perceived that the living environment was inap- 
propriate, second, those who could communicate, third, those with physical disabilities 
registered in the Disabled Welfare Act, fourth, those who did not plan to move within 
three months, fifth, understood the study and agreed to participate. From February 2022 
to April 2022, we visited five participants at home and provided SHbHM. The charac- 
teristics of the participants were 4 men and 1 woman, with an average age of 66.6 years, 
3 strokes, and 2 spinal cord injuries. 


2.2 Independent Variable (Smart Home-based Home Modification; SHbHM) 


This study used the SHbHM program to improve the occupational performance and 
quality of life of PwPD. We did the occupational profile to the participants. Begin- 
ner Kits or expert kits were provided according to the occupation required through the 
occupation profile. After the kit is provided, the usage log for the kit is automatically 
logged. Occupational therapists monitor usability through participant logs, provide task- 
oriented and simplify occupational performance skills. In the SHBHM, the occupational 
therapist visited the subjects a total of eight times, including one initial assessment, one 
occupational profile, one kit installation and usability assessment, five task-oriented and 
simplify occupational performance skills, and one follow-up assessment (Fig. 1). 
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Fig. 1. Smart home-based home modification 
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Occupational Profile. The occupational profile is summary of a participant’s occupa- 
tional history and experiences, patterns of daily living, interests, values, needs, and Smart 
Home relevant contexts [9]. Using a client-centered approach, the occupational therapist 
collected information to understand what participants want and need through SHT and to 
identify past experiences and interests that can help them understand current problems. 
The researcher evaluated the participants’ occupational performance, satisfaction, and 
health-related quality of life before and after SHbHM. Canadian Occupational Perfor- 
mance Measure (COPM) was used for occupational performance and satisfaction [10], 
and EuroQol-5 dimension (EQ-5D) was used for health-related quality of life [11]. 


Beginner Kit. PwPD has restrictions on hand use and indoor walking. It is difficult to 
control the lighting and unlock the door in life. Therefore, SHT is required. Even without 
smartphones and Wi-Fi, Participants can apply The Beginner kits at low cost without 
continuous cost through Bluetooth. The researcher monitored the participants’ use logs 
for 4 weeks. 


Expert Kit. The expert kit consists of door sensors, motion sensors, smart switches, 
Zigbee hubs, electric curtains, smart bulbs, AI speakers, smart plugs, and smart remote 
controls. It was automated according to the participants’ daily lives. The use of Wi-Fi and 
smartphones was essential to use expert kit. The researcher monitored the participants’ 
use logs for 4 weeks. 


Task-oriented and Simplify Occupational Performance Skills. As The Task- 
Oriented focused on special functional tasks that incorporate musculoskeletal or ner- 
vous systems [12], we provided task-oriented routines rather than normal pattern 
repetitive exercises participants learned and learned skills by simplifying occupational 
performance skills [13]. 


2.3 Dependent Variable 


Changes in the Quality of Occupational Performance. We measured the quality of 
occupational performance evaluating the canadian occupational performance measure 
(COPM). Originally published in 1991, COPM is a customer-centric, individualized 
measure of results. Importance, performance, and satisfaction are evaluated on a 10- 
point scale according to occupation, and changes in scores after pre-score can be seen as 
therapeutic effects, which are clinically useful if there is a difference of more than two 
points [10]. According to the COPM guidelines, test-retest reliability has been reported in 
three studies, with performance reliability of 0.65, 7.99, 0.80, and satisfaction reliability 
of 0.84, 7.75, and 0.89, respectively [14] The evaluation was conducted before and after 
SHbHM. 


Changes in the Quality of Life. Participants' quality of life was measured through 
EuroQol-5 Dimension (EQ-5D). EQ-5D, developed by EuroQol Group, is an indicator 
of health-related quality of life and can be used to evaluate health-related quality of life for 
people with disabilities [11]. EQ-5D is a total of five areas: mobility, self-management, 
daily activities, pain/comfort, and anxiety/depression. It is composed of 1 to —1 by 
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applying weights to each measurement, and the overall health condition is —1 which 
is worse than 1 point death. The health-related quality of life score was calculated by 
applying the weight calculation equation according to Lee et al. [15]. The evaluation 
was conducted before and after SHbHM. 


3 Results 


Our experiment used the SwitchBot platform for the Beginner Kit, and the Expert Kit 
used SmartThings, SwitchBot, AI power manager platform for log verification. The 
participants’ usage log was checked by the researcher through IDE (https://graph.api. 
smartthings.com/) and application every day and recorded in Excel for 30 days (Fig. 2). 
As can be seen in Fig. 2, Expert Kit showed an increase of 500% from a 7 use per day 
to 35 use per 30 days after SHbHM was applied. 


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 


ee Beginner Kit —9— Expert Kit 


Fig. 2. SHbHM usage logs 


We evaluated participants' occupational performance, satisfaction, and health- 
related quality of life before and after SHbHM. The results of the five are as follows 
(Table 1). As can be seen in Table 1, Participants increased occupational performance 
6.2 (SD 1.7), occupational satisfaction 5.3 (SD 2.5), and quality of life 0.302 (SD 0.075). 


Table 1. COPM, EQ-5D change scores for participants who had a SHbHM. 


Evaluation M (SD) 
Initial assessment (n — 5) | Reassessment (n — 5) | Change (n — 5) 
COPM | Performance | 2.7 (1.6) 8.9 (1.2) 6.2 (1.7) 
Satisfaction 3.0 (1.5) 8.3 (2.1) 5.3 (2.5) 


EQ-5D | Quality of life | 0.474 (0.074) 0.776 (0.036) 0.302 (0.075) 
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4 Conclusion 


This study bridged the gap in a pilot study on the service delivery model used to provide 
SHbHM as a home modification and confirmed the need for future studies. In addition, 
the purpose of this pilot study was to develop a new program and provide it to PwPD 
to find out the applicability of the technology. However, there is a limitation in that the 
sample size is small to verify the effect of the applied SHbHM. In future studies, it plans 
to verify the effectiveness by expanding the number of subjects. although the number of 
research subjects was small, it was significant in that it was a study that applied smart 
home technology to the homes of the PwPD in Korea. 

The results of this study were intervention and home modification that enable home 
management for PwPD. The provision of SHbHM has the potential to inform the devel- 
opment of cost-effective service delivery models. The use of SHbHM as home modifi- 
cation to improve PwPD's occupational performance and health-related quality of life is 
extending area of research and intervention used in practice which is eventually tailored 
in community for occupational therapist. 
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Abstract. Wearing a mask is an effective measure that prevents the 
spread of respiratory droplets into the air and thereby curtails the dis- 
semination of coronavirus. Unfortunately, despite the proven effective- 
ness, the idea of wearing a face mask has difficulty being accepted by 
part of the population. To address this significant health concern, we 
present a monitoring system that automatically detects whether a mask 
is put appropriately over a face. The system annotates the videos that are 
provided by cameras. In this article, we present a comparative study of 
machine learning models (i.e., SVM, RNN, LSTM, CNN, auto-encoder, 
MobileNetV2, Net-B3, VGG-16, VGG-19, Resnet-152). 


Keywords: COVID-19 - Face mask detection - IoT 


1 Introduction 


While there are multiple ways to fight the pandemic, a face mask remains a 
cost-effective measure that is widely practiced to prevent the spread of the 
virus. Nonetheless, the adoption of face-mask measures remains controversial. 
The ability to detect violations in public and work spaces is of utmost impor- 
tance for organizations that host a large population and wish to monitor expo- 
sure to adopt precautionary measures. In this article, we propose a detection 
system that determines whether a person is appropriately wearing a mask, using 
the information provided by some cameras. Cameras may be disseminated in 
a building to provide some videos (i.e., image sequences) that are further used 
to support face mask recognition and ultimately issue proper directives. Still, 
detecting face masks in any situation is challenging because there are many 
variations in the appearance of the image that may alter the detection (Fig. 1): 
variations can be due to (i) a disruptive context (e.g. occlusion of the person, low 
light intensity), (ii) a rotation or an poor orientation of the person that hinders 
the detection, (iii) camera parameters (low resolution, excessive focus, noise) or 
poor positioning of the camera (misorientation, large distance to the scene). We 
© The Author(s) 2022 
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(a) One person of the three wears a mask (b) Two persons wear a mask ; one per- 
son has sun glasses 


(c) one mask is incorrectly put, a(d) 2 persons in profile ; 2 persons wear a mask and 
person has glasses one in front wears a cap 


Fig. 1. Situations in which real-time videos are automatically annotated 


propose a comparative study of different ML models and evaluate their ability to 
handle the above situations and their suitability for deployment on IoT devices 
or cloud servers. Overall, our key contribution includes: 


— A face mask detection service that exploits videos. 

— A comparative study of various ML models (i.e., CNN, LSTM, RNN, auto- 
encoder, SVM, MobileNet V2, Net-B3, VGG-16, VGG-19, Resnet-152) with a 
discussion on their practical applicability. 

— A prototype that annotates the collected videos relying on a IoT device or on 
a remote (cloud) server ; the choice of the approach is based on requirements 
of the organization. 


The paper is organized as follows. We present the related work (Sect. 2) and we 
introduce our comparative study (Sect. 3), which is further evaluated (Sect. 4). 


2 Related Work 


Several detection systems [5-7,10,11,14,16,17] have been proposed and evalu- 
ated using some purposely-built datasets (Table 1) that contain some pictures of 
people with facemask, without facemask or with facemask put incorrectly. The 
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pictures of masked people either (i) correspond to real-world person that are pic- 
tured (real-world dataset) or (ii) result from the addition of a mask picture on an 
existing facial image (simulated dataset). Leveraging existing datasets, detection 
system detects facemasks on pictures (Sect. 2.1) or videos (Sect. 2.2). 


Table 1. Datasets containing real world images or simulated images 


Dataset Size Content Ref 


Real world dataset 


Real World Masked Face Reco. Dataset:RMFRD | 14 K | 0.5 K public figures with/without mask 4 
Masked Face Detection Dataset: MFDD 24 K | persons with mask 4 
Face Mask Dataset: FMD 0.8 K | with/without mask, mask put incorrectly 2 
Medical Masks Dataset: MMD 6K 3 k medical masked faces 3 


Simulated dataset 


Simulated Masked Face Recog. Dataset: SMFRD | 500 K | simulated facial images, 10 K participants | [4 


Custom Mask Community Dataset (CMCD) 1.2 K | with/without simulated facemask T 


2.1 Facemask Detection in Pictures 


The facemask detection system introduced in [11], consists of a feature extractor 
that uses convolutional neural network (Resnet50!) and a classifier that imple- 
ments Support Vector Machine (SVM) and ensemble algorithm. Evaluations 
based on the RMFD, SMFD, CMCD datasets, show that the SVM classifier 
involves the fastest training and achieves the highest accuracy: 99.64% test- 
ing accuracy with RMFRD, 99.49% with SMFD, and, 100% with CMCD. In 
[10], the solution localizes medical face masks and annotates accordingly those 
images. Feature extraction process relies on the ResNet50 deep transfer learning 
model while the mask detection process uses YOLO v2 [12]. Following, authors 
rely on the Adam optimization algorithm [8] to improve the performance of the 
detector. Empirical evaluation shows that the Adam optimizer achieves the high- 
est average precision percentage of 81%. In [14], face mask detection uses the 
faceNet image classifier [15] that implements a Convolutional Neural Network 
(CNN). This image classifier is trained using a purposely-built dataset including 
4K images with half of the dataset containing some pictures of people wearing 
mask in public places (e.g. shops) while the rest concerns people without mask. 
Empirical results show that people wearing (or not) a face mask are detected 
with an accuracy of 96,8596. Arjya et al.[6] detect the facemask on image using 
a pre-trained CNN containing two 2D convolution layers connected to layers 
of dense neurons. The proposed method attains an accuracy up to 95.77% with 
SFC dataset and 94.58% respectively FMD dataset. In [7], a facial categorization 
system determines whether a person is wearing a mask or not. Face recognition 
is performed by a deep C2D CNN (Colour 2-Dimensional principal component 
analysis - Convolutional Neural Network) ; mask detection relies on special con- 
volutional architecture that is best suited for the classification of RGB images. 


1 https: //keras.io/api/applications/resnet. 
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The training relies on the RMFRD and Celeb Faces Attributes? dataset. In [5], 
the face mask detection system captures image, extracts features from image 
based on Principal Component Analysis (PCA), detects the human face using 
viola zones method and further uses the K-Nearest Neighbor (KNN) classifier. 
Experiments are based on the ORL? database in which the lower portion of 
detected face images is covered with black or white. Preliminary performance 
evaluation shows that the accuracy is around 98% with a principal component of 
two. Additionally, face recognition accuracy with face masks has been extensively 
investigated: interest reader may refer to the masked face recognition workshop 
and challenge’. 


2.2 Face Mask Detection in Video 


Another line of research aims at providing a surveillance system [17] that iden- 
tifies whether a person is wearing a mask using real-time videos. Mask detection 
is done by MobileNetV2 [13] that achieves high accuracy of 99.98% on training 
data, 99.56% on validation data, and 99.75% on testing data. In [16], a mobile 
robot automatically detects unmasked personnel in public spaces and provides 
a surgical mask to them to promptly remedy the situation. The mobile robot 
integrates deep residual learning (ResNet-50) with Feature Pyramid Network 
(FPN) to detect the existence of human subjects in video (feeds). Then, Multi- 
Task Convolutional Neural Networks (MT-CNN) detect and extract human faces 
from these videos. Ultimately, a convolutional neural network classifier detects 
(un)masked human subjects. Training leverages four publicly available datasets: 
Microsoft Common Objects in Context (COCOJ)[9], the CelebFaces Attributes 
Dataset? (CelebA), WIDER FACE dataset, CMCD. The proposed surveillance 
system is further evaluated using a dataset of videos collected by the robot 
in an educational institute. Results show a mask detection accuracy of 81.3% 
with a very high recall of 99.2%. While many detectors rely on pictures, only 
two approaches support real-time facemask detection leveraging videos. In this 
paper, we introduce a video-based system that incorporates several ML models 
and we provide a comprehensive comparison with the state of the art. 


3 Mask Detection 


Leveraging the videos delivered by the camera, our application detects the pres- 
ence of any nearby person and determines whether the person has a mask and 
if the mask is correctly put. Then, the application labels the corresponding 
image. This detection requires converting the videos into an appropriate format, 
locating people face(s) (Sect. 3.1) and determining whether people wears mask 
(Sect. 3.2). 


? https:/ /www.kaggle.com/jessicali9530/celeba-dataset. 
3 https:/ /cam-orl.co.uk/facedatabase.html. 

^ https: //tinyurl.com/4xzjzvat. 

5 https://www.kagele.com/jessicali9530/celeba- dataset. 
6 https: //paperswithcode.com/dataset /wider-face-1. 
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3.1 Face Detection in Pre-processed Video 


Face mask detection starts with the capture of a video partitioned in successive 
series of color pictures. Color pictures are further converted into RGB pictures, 
which render the process of discovering the face less complex comparing to color 
picture. For each picture, mean subtraction is applied to prevent illumination: 
the average intensity is computed accross all the images for each of the Red, 
Green, and Blue channel ; then, the mean is substracted per channel for each 
image. Following, each image is divided into n x n squares where the value of 
n depends of the object of interest (e.g. main feature of the face). Within each 
square, the face detection algorithm passes through each pixel of the image in 
addition to the adjacent pixels (i.e. the pixels located at the top - bottom - left 
- right - top right - top left - bottom right - and bottom left). This process is 
intended to store key facial features (e.g. eyes) that help in detecting face while 
irrelevant data that are located in the background (e.g. a car, tree, traffic light). 
Finally, our system determines the location of each face. Our approach consists 
in classifying into two classes, whether it is wearing appropriately a facemask or 
not. Note that the facemask is appropriately wear if the mouth, chin and part 
of the nose are well covered with the mask. 


3.2 Classification 


For facemask detection, we built some models using SVM, RNN, LSTM, Auto- 
Encoder, which are briefly presented, starting with SVM. 


Support Vector Machines SVM separates the input data within the space by a 
hyperplane that linearly separates the data into classes (with and without mask). 
Input data typically refers to small training dataset made of support vectors. 
Herein we use a linear kernel. The hyperplane best separates the support vectors, 
by means of maximization of the distance between these vectors and the hyper- 
plane. As shown in Sect. 4, the SVM remains efficient with little training data. 

In addition, we consider three types of Convolutional Neural Networks (CNN) 
referring to Recurrent Neural Networks (RNN) and Long Short Term Memory 
Networks (LSTM) - that are multi-layered neural networks made of several hid- 
den layers of neurons wherein the output of a neuron in a layer becomes the 
input of a neuron of the next layer. These networks have adopted diverse struc- 
tures that meet different expectations. Convolutional Neural Network (CNN) is 
a neural made of two distinct parts: (i) the convolution layers extracts valuable 
features from the input (image); in practice, kernels automatically extract the 
relevant features based on the convolution operation, (ii) the fully connected 
layers leverage the data from convolution layer to generate the result. 


Recurrent Neural Network (RNN) axe a class of neural networks that differs from 
others in that they maintain internal hidden states and have cyclic/recurrent 
connections, which allow them (i) to capture the sequential information (i.e. 
dependencies) in the input data and (ii) information to persist. Still, RNN tra- 
ditionally suffers from what is known as the problem of vanishing and exploding 
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gradient in which the network either stops learning (vanishing gradient) or never 
converges to the point of minimum cost (exploding gradient). LSTM are designed 
to remedy both problems and thereby have become popular in modelling com- 
plex sequential data. 


Long Short Term Memory Networks consists of a set of recurrently connected 
subnetworks (also coined as memory blocks). Any block contains one or more 
self-connected memory cells storing historical states, as well as gates that control 
the flow of information through the cells. Thus, LSTM may store and access 
information over long period of time, which prevents the vanishing gradient 
problem. LSTM contains four layers of neural networks. 


Auto Encoder is a specific type of neural network in which the encoder represents 
the input into a compressed and meaningful representation so that the decoder 
has the most relevant information to reconstruct the image. In particular, the 
encoder learns the most important components of an input and thereby gets the 
best possible compression. The error made by the encoder is established based on 
the differences between the reconstructed data and the initial data. The training 
consists in modifying the parameters of the auto-encoder so as to reduce the 
reconstruction error measured on the different samples of the dataset. While 
various neural network topologies exist (e.g., vanilla, convolutional, regularized, 
multi-layer), we used a multi-layer auto-encoder and we encoded in an unsuper- 
vised way. The encoder contains three hidden layers: the first one is four times 
larger than the input, and the second one is two times larger than the input, 
and the size of the third one is equal to the input size. Following, we optimize 
the model using adam optimiser [8]. 


4 Performance Evaluation 


We assess the effectiveness of our detector relying on the following two training 
datasets: the Real World Masked Face Recognision Dataset [4] and the Face 
Mask Dataset [2] (FMD) that include some color pictures of different sizes. 
Together these two datasets include some pictures of people of different nation- 
alities/ages, with/without facemask, with mask put incorrectly, with e.g., glasses, 
hat. As detailed in Sect. 3.1, pictures are normalized and expressed into a com- 
mon format. We performed cross-validation, using scikit-learn platform", which 
splits the dataset into a training (8096 of the original dataset) and test dataset. 
Unless explicitly mentioned, 100 epochs are for building models. In the following, 
we evaluate the performances associated with the training and the detection. In 
both cases, the experiments are run either on a IoT device (Raspberry Pi 4 with 
1,6 GHZ and 2 GB of RAM) with 3,1 GHZ of CPU, and Machine ASUS with 
8 GB in RAM, used to perform the training and the mask detection. In order 
to evaluate the detection of facemask, we also conducted series of experiments 
using dataset and a camera of 48 Mega pixel. 


T https:/ /scikit-learn.org. 
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Table 2. Size of dataset used to train and test the model 


Size of dataset Small | Medium | Large 
Training 1000 | 2000 6000 
Testing | 200 | 400 1200 


Delay associated with the training and detection. In Table3, we evaluate the time 
associated with the learning process, using various sizes of dataset (Table 2). As 
expected, the larger the dataset, the longer the learning process. The device 
capacity also has a significant impact on its ability to learn (quickly). Contrary 
to the server, the IoT device can only handle small or medium datasets, regard- 
less of the learning model. The learning process is on average 2.631 times longer 
with the IoT device whose capabilities are limited, compared to the server. On 
the other hand, detection is much faster than learning (Table 4). In particular, 
the time associated with detection is completely decoupled from that associ- 
ated with learning. The detection with the Raspberry takes a little longer than 
with a server. For both (IoT device an server), CNN is the fastest, followed by 
autoencoder, LSTM, RNN whose results are close while SVM is the slowest. 


Table 3. Training Delay associated with a dataset of varying size, using an IOT 
device and a server 


Dataset | Device SVM RNN LSTM CNN AUTOENCODER 
Small Raspberry | 13 h20 mn |14 h35 mn | 15 h30 mn | 13 h07 mn | 15 h20 mn 

Small Server 5hl0mn |5hl15 mn |4h50mn |5h20mn |6h05 mn 
Medium | Raspberry | — — — = — 

Medium | Server 6h50mn |7h30mn |8h04mn |8 h30 mn |10 h15 mn 

Large Raspberry | — = = = m 

Large Server 14 h30 mn |13 h50 mn | 14 h 10 mn | 15 h25 mn | 16 h 30 mn 


Table 4. Delay associated with detection 


Model SVM | RNN | LSTM | CNN | AUTOENCODER 
Raspberry |4s /|2.8s | 2.6s 1.5s |2.28 
Server 3s 1.9s |1.5s |0.8s|l4s 


Facemask Detection Efficiency We compare the effectiveness of four strategies 
used to detect face masks (as defined in Sect. 3). Experiments were run on the 
IoT device and on the server; results reported bellow are the same for both. 
Figure 2 provides the accuracy and the loss functions associated with the train- 
ing (accuracy and loss) and the validation (val_accuracy, val_loss) sets; the 
training set is used to build the model while the validation set supports the 
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Fig.2. Accuracy and Loss associated with the training and validation with large 
dataset 


'Table 5. Dataset size 


Class Small dataset | Medium dataset | Large Dataset 
With mask | 1000 2000 4718 
With mask | 1000 2000 1759 


fine tuning of the parameters and of the model structure. Loss corresponds to 
binary-cross entropy. For the RNN, LSTM and CNN models, accuracy and loss 
appear to be inversely proportional: after few iterations of optimization, the loss 
reduces drastically while the accuracy greatly increases. As intended, the shape 
of accuracy and loss functions are quite similar with the training and validation 
sets. At first sight, CNN becomes quickly accurate (accuracy is high and loss is 
small after only 40 epochs). With LSTM (and resp. RNN), the accuracy and loss 
stabilize after 60 epoch (resp. 80 epochs). The auto-encoder takes much more 
time (400 epochs) comparing to the other methods: the loss curve still fluctuates 
for epoch ranges of [200,250] and [300,350]. In addition, we evaluate the per- 
formances associated with the detection in terms of precision, recall, F1 score 
and accuracy. Relying on these performance metrics, we evaluate in Tables 6, 7 
and 8 the efficiency of the detection models considering a small, medium and 
large dataset (as defined in Table5). As expected, the larger the dataset, the 
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Table 6. Models with small dataset 


Algo Precision | Recall | F1-score | Accuracy 
SVM Without mask | 0.83 0.90 0.85 0.84 
With mask 0.84 0.83 0.92 
Macro avg 0.82 0.83 082 
Weighted avg | 0.84 0.84 0.84 
RNN Without mask | 0.85 0.84 0.84 0.85 
With mask 0.86 0.85 0.84 
Macro avg 0.86 0.85 0.86 
Weighted avg | 0.86 0.86 0.86 
LSTM Without mask | 0.89 0.84 0.85 0.88 
With mask 0.89 0.87 0.86 0.86 
Macro avg 0.86 0.86 0.85 
Weighted avg | 0.87 0.87 0.87 
CNN Without mask | 0.93 0.93 0.4 0.95 
With mask 0.96 0.97 0.95 
Macro avg 0.94 0.95 0.95 
Weighted avg | 0.95 0.95 0.95 
Auto encoder | Without mask | 0.88 0.87 0.85 0.91 
With mask 0.90 0.88 0.86 
Macro avg 0.89 0.88 0.86 
Weighted avg | 0.90 0.89 0.87 


Table 7. Efficiency of learning models with medium dataset 


Model Precision | Recall | F1-score | Accuracy 
SVM Without Mask | 0.85 0.86 0.86 0.87 
With Mask 0.86 0.85 0.84 
Macro Avg 0.86 0.87 0.85 
Weighted Avg | 0.87 0.87 0.87 
RNN Without Mask | 0.87 0.85 0.88 0.89 
With Mask 0.89 0.90 0.90 
Macro Avg 0.90 0.89 0.90 
Weighted Avg | 0.90 0.90 0.90 
LSTM Without Mask | 0.89 0.88 0.90 0.91 
With Mask 0.91 0.91 0.91 
Macro Avg 0.90 0.89 0.91 
Weighted Avg | 0.91 0.91 0.91 
CNN Without Mask | 0.91 0.91 0.92 0.97 
With Mask 0.98 0.95 0.96 
Macro Avg 0.95 0.96 0.96 
Weighted Avg | 0.97 0.96 0.97 
Auto Encoder | Without Mask | 0.91 0.90 0.92 0.94 
With Mask 0.92 0.93 0.92 
Macro Avg 0.91 0.92 0.92 
Weighted Avg | 0.92 0.93 0.92 
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Table 8. Efficiency of models with large dataset 


Model Precision | Recall | F1-score | Accuracy 
SVM Without Mask | 0.86 0.89 0.88 0.89 
With Mask 0.91 0.89 0.90 
Macro Avg 0.88 0.88 0.89 
Weighted Avg | 0.89 0.88 0.89 
RNN Without Mask | 0.91 0.90 0.91 0.90 
With Mask 0.93 0.92 0.92 
Macro Avg 0.93 0.91 0.93 
Weighted Avg | 0.94 0.94 0.94 
LSTM Without Mask | 0.94 0.91 0.92 0.96 
With Mask 0.97 0.95 0.96 
Macro Avg 0.95 0.94 0.95 
Weighted Avg | 0.96 0.96 0.96 
CNN Without Mask | 0.98 0.99 0.99 0.99 
With Mask 0.99 0.99 0.99 
Macro Avg 0.98 0.98 0.98 
Weighted Avg | 0.99 0.99 0.99 
Auto Encoder | Without Mask | 0.91 0.92 0.92 0.94 
With Mask 0.92 0.93 0.92 
Macro Avg 0.93 0.92 0.92 
Weighted Avg | 0.94 0.94 0.94 


Table 9. Efficiency of state of the art models using dataset3 


Algo Precision | recall | F1-score | Accuracy 
MobileNet V2 Without Mask | 0.95 0.94 | 0.95 0.96 
With Mask 0.96 0.95 | 0.96 
Macro Avg 0.96 0.95 | 0.95 
Weighted Avg | 0.96 0.96 | 0.96 
Efficient Net-B3 | Without Mask | 0.93 0.92 | 0.91 0.94 
With Mask 0.93 0.92 | 0.92 
Macro Avg 0.93 0.92 | 0.91 
Weighted Avg | 0.93 0.93 | 0.92 
VGG-16 Without Mask | 0.96 0.95 | 0.97 0.98 
With Mask 0.98 0.98 | 0.96 
Macro Avg 0.97 0.97 | 0.96 
Weighted Avg | 0.98 0.97 | 0.97 
VGG-19 Without Mask | 0.96 0.96 | 0.92 0.98 
With Mask 0.98 0.95 | 0.96 
Macro Avg 0.97 0.95 | 0.94 
Weighted Avg | 0.98 0.96 | 0.94 
ResNet-152 Without Mask | 0.98 0.99 | 0.99 0.98 
With Mask 0.99 0.99 | 0.98 
Macro Avg 0.98 0.98 | 0.98 
Weighted Avg | 0.99 0.99 | 0.99 
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more effective the training and subsequent detection, as shown by the increase 
in precision, recall, F1 score and accuracy for any model. Note that data augmen- 
tation may be relevant to increase the dataset size (and thereby the efficiency 
of the ML models) by creating modified versions of images. Regardless of the 
dataset size and of the model, the accuracy associated with the detection of 
people with mask is better than that of people without mask. Regardless of the 
dataset size, CNN always gives the best result in terms of precision, recall, F1 
score, accuracy. Then, LSTM and auto-encoder provide lower but high efficiency, 
followed by RNN and SVM. Table 9 compares various models. CNN gives the 
best results in terms of accuracy while ResNet-152 is characterised by a slighly 
lower accuracy but a quite similar precision, recall and F1-score. 


5 Conclusion 


We introduce a new detection system that automatically determines whether 
a person wears facemask, which is put appropriately. In practice, the system 
detects and determines the position of the face(s) in the videos provided by 
cameras. For each detected face, the system determines whether a facemask 
is put appropriately, leveraging some machine learning models. Experimental 
results show that classification may be performed by a server or IoT device. 
Empirical results demonstrated that CNN gives is the most accurate (99.8% 
with a large training dataset). Future work involves improving the detection in 
presence of low quality picture (e.g., low light level, presence of obstacles) and 
evaluating the energy associated with the detection. 
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Abstract. An integral part in the comprehensive care of HIV and a significant 
health service is contraception, however research carried out to evaluate the per- 
ception and utilization of contraception among HIV positive men are few. This 
research aim to determine the knowledge, perception and uptake of contraception 
among HIV positive male patients at Saye-Zaria. This was a descriptive, cross- 
sectional study with collection of quantitative data through questionnaire and 
qualitative data with Focused group discussion (FGD). The majority (85.196) of 
respondents have heard about contraception, most had good perception, and only 
61.996 of the respondents have ever used contraception. There was a significant 
association between level of education and perception. In conclusion there was 
high knowledge, low usage and poor acceptance of contraception. Therefore, the 
government should put adequate policies in place to encourage male involvement 
in the utilization of contraception. 


Keywords: Male patient - HIV - Nigeria - Contraception - Perception 


1 Introduction 


Nigeria is second to South Africa in the number of people living with HIV/AIDS world- 
wide, representing 9% of the global burden of the disease [1] while men account for 58% 
of adult AIDS related deaths [2]. HIV is mainly transmitted through heterosexual con- 
tact and new infections in the country heighten due to reduced perceived personal risk, 
multiple sexual partners, inefficient and inadequate treatment of sexually transmitted 
infections (STIs) and poor quality service delivery [1]. 

In Nigeria, the total contraceptive prevalence rate (CPR) indicates wide state varia- 
tions, ranging from 0.346 in Jigawa to 41.696 in Lagos state, as well as zonal variations 
ranging from 2.7% in the North West to 28.5% in the South West [3]. In a study done 
in 2003 among monogamous men in Ondo State Nigeria, the study revealed that 81.096 
of the respondents knew of condom, only 31.2% had ever used a condom and 15.0% 
reported currently using condoms at the time of interview [4]. This trend was also seen 
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in a study done in 2006 on male knowledge, attitudes, and family planning practices 
in northern Nigeria where the results suggested that there is high knowledge of con- 
traceptives and consequently low rates of contraceptive use [5] of which the reasons 
might range from cultural to religious misconceptions about men using contraception. 
There are overwhelming previous studies carried out to assess the uptake, utilization 
and factors affecting contraceptive use among HIV positive women and prevention of 
mother to child transmission of HIV (PMTCT) in both urban and low resource settings. 
However, research carried out to evaluate the perception and utilization of HIV posi- 
tive men are few [5, 6] although it is the men that are usually the decision makers on 
important household issues like household purchase, family size, health of household 
members and timing of pregnancy in this country [7]. The uptake of contraceptives and 
the type of contraceptive use is heavily influenced by the male/husband dominance in 
the society/family. Therefore, there is a need to integrate and involve the male in the 
contraception policies [8]. 

In this study, we evaluated the knowledge, perception and uptake of contraception 
among HIV positive male patients in Nigeria. We majorly focused our study on patients 
attending ART clinic of National Tuberculosis and Leprosy Training Center (NTBLTC) 
Saye-Zaria. So, we provided answers to the following questions: (1) What is the knowl- 
edge of contraception among HIV positive male patients? (2) What is the perception of 
contraception use among HIV positive male patients? (3) Whatis the use of contraception 
among HIV positive male patients? (4) What are the factors influencing contraceptive 
uptake among HIV positive male patients? 


2 Methodology 


The study was a descriptive, Cross-sectional study collecting both qualitative and quan- 
titative data. The study population was HIV positive male patients receiving treatment at 
the ART clinic at NTBLTC Saye-Zaria, Nigeria. Inclusion criteria was all HIV positive 
male patients attending ART clinic, HIV positive male patients who are >18 years and 
HIV positive male patients who are taking ART at the time of the interview. The sample 
size was determined using the formula [9] 


Z*pq 

n= az (1) 

A sample size of 285 was used as the minimum sample size for this study. A simple 
random sampling technique was used to collect information from eligible participants. 
The data was collected using adapted, semi-structured and pre-coded questionnaires 
which were collected via an android device using KoBo kollect software and were admin- 
istered to each respondent. The questionnaire contained information on respondent’s 
sociodemographic status, knowledge of contraception, perception of contraception use, 
uptake of contraception and factors influencing the use of contraception. The knowledge 
of contraception was based on those that have ever heard of contraception before. The 
perception on contraception was scored on a 2-point perception scale for each question, 
positive perception was 2 point, negative perception was 0 points and undecided was 1 
point. Those that score less than or equal to one (<1) point had negative perception and 
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those that score greater than one (>1) point had positive perception. The quantitative 
data collected was entered and analyzed using STATA, univariate analysis was done 
using proportions, measures of dispersion, measures of central tendency and percentage 
while bivariate analysis using chi square was done. A confidence interval of 95% was 
used and a p value of <0.05 was considered statistically significant. 


3 Results and Discussion 


3.1 Socio-Demographic Characteristics 


The study (in Fig. 1) showed that 84 (31.396) of the respondents were within the 38—47 
age group, the mean age of respondents was 45.6 + 11.7. Majority of the respondents 
(81.0%) were Muslims, 50 (18.7%) were Christians, most of the respondents were of 
Hausa tribe 194 (72.4%), Fulani/Ibo (7.8%), Yoruba/Others (6%). Although, 79.1% of 
the respondents were married, 147 (69.3%) are monogamous and 30.0% have more than 
one sexual partner. 
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Fig. 1. The socio-demographic characteristics of respondents. 


Some (27.296) of the respondents have at least secondary education, 2596 have 
Quranic education, 20.2% have tertiary education, 14.9% post-secondary and 3% have 
no formal education. 57 (21.396) of the respondents are traders and self-employed, 54 
(20.2%) are civil servants, 47 (17.5%) were farmers and only 6 (2.2%) are unemployed. 
More than half of the respondents 146 (54.596) reside in urban areas while 122 (45.596) 
reside in rural areas. 


3.2 Knowledge of Contraception Among HIV Positive Male Patients 


Majority 228 (85.196) of the respondents have heard about contraception before which 
showed that there is good knowledge about contraception and almost all 206 (90.496) 
of the respondents knew that condoms are a type of contraception, as shown in Fig. 2. It 
is also similar to a study done in Uganda Rhoda and a study done in Zimbabwe where 
more than 9846 of the men reported that they have heard of at least one method of 
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contraception [10, 11]. However, it is in contrast to a study done in Osogbo-Nigeria on 
male involvement in family planning where only 57.0% had a good knowledge of FP 


[3]. 
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Fig. 2. Knowledge of respondents about contraception. 


Other types of contraception identified by the respondents were Pills 48 (21.1%), 
Injectable 41(18%), Withdrawal 24 (10.5%), only 15 (6.6%) and 10 (4.4%) of the respon- 
dents knew Male sterilization and Female sterilization respectively. The least known 
methods were Foam or Jelly (0.4%), Emergency contraception (0.4%), LAM 2 (0.9%), 
Diaphragm 3 (1.3%), IUD 6(2.6%) and Traditional method 8 (3.5%). Mostly half of 
the respondents knew that contraception is used for Prevention of pregnancy and Pre- 
venting the transmission of HIV/STIs, while 28 (12.396) of the respondents think that 
contraception is neither useful for both. 

Half (50.4%) of the respondents in this study got their information about con- 
traception from ART clinic/Hospital, while 30.7% got their information from radio. 
This is followed by 796 of respondents that got their information from television and 
friends/family. 


3.3 Perception of Contraception Among HIV Positive Male Patients 


The overall mean perception score (in Table 1) of the respondents was 1.38 + 0.52 
and most of the respondents 192 (72%) had good perception while 76 (28%) had poor 
perception. 22.096 were of the perception that it is only promiscuous women that use 
contraception without their husband's consent, 23.596 of the respondents were of the 
perception that they will not allow their spouse to use contraception, the perception of 
15.796 of the respondents was that it is only women that are meant to use contraception 
and not the men. 
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Table 1. Perception of respondents on contraception. 


Statements Disagree (%) | Not sure (%) | Agree (%) 
It is only women who are promiscuous that use | 127 (47.4)* 82 (30.6) 59 (22.0) 
contraception without their husband’s consent 

I will not allow my spouse/partner to use 165 (61.6)* 40 (14.9) 63 (23.5) 
contraception 

It is only women that are meant to use 167 (62.3)* 59 (22.0) 42 (15.7) 
contraception and not the men 

There is no need for HIV positive man to use 160 (63.1)* 55 (20.5) 44 (16.4) 
contraception 

Contraception should not be promoted among 153 (57.1)* 56 (20.9) 59 (22.0) 
people living with HIV 

Men should not assist their women in obtaining | 156 (58.2)* 60 (22.4) 52 (19.4) 
contraception 


“Positive responses 


About 16.4% and 22.0% perceived that there is no need for HIV positive men to use 
contraception and contraception should not be promoted among people living with HIV, 
while the perception of 19.4% of the respondents was that men should not assist their 
women in obtaining contraception this is in contrast with the study done at secondary 
health facility in North-Central Nigeria where 97.8% of the respondents said condom 
use should be promoted among people living with HIV/AIDS [12]. 


3.4 Uptake of Contraception Among HIV Positive Male Patients 


Out of the 268 respondents (in Table 2), many of the respondents 166 (61.9%) reported 
that they have ever used contraception and 91 (34%) reported that their spouse has ever 
used contraception before while only 152 (56.7%) are currently using contraception 
which is almost synonymous to a study done in Nairobi, Kenya where 58.8% of the 
male respondents had used contraception [13]. 

Out of 166 respondents that reported that they had ever used contraception, 153 
(92.2%) chose condom as one of the contraceptive methods that they have ever used, 17 
(10.2%) withdrawal method, 7 (4.2%) abstinence, 4 (2.4%) traditional method, 3 (1.8%) 
chose pills while only 2 (1.2%) chose male sterilization. 152 respondents reported that 
they are currently using contraception of which condom still ranked highest contracep- 
tive method used at 138 (90.8%), the second was withdrawal 14 (9.2%), followed by 
injectable at 8 (5.3%) and male sterilization at 3 (2.0%). None of the respondents chose 
IUD, LAM, foam/jelly, diaphragm and emergency contraception as a method they have 
used or currently using. 

About 65 (39%) of the 166 respondents that have ever used contraception reported 
that the reason for their contraceptive use was to prevent pregnancy, 59 (36%) stated 
prevention of re-infection as the reason while 42 (25%) stated prevention of re-infection 
and pregnancy as the reason for contraceptive use. 
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Table 2. Contraceptive methods ever used and currently using by respondents. 


Contraceptive method Ever used contraceptive (%) n = | Currently using contraceptive 
166 (%) n = 152 
Condom 153 (92.2)* 138 (90.8)* 
Withdrawal 17 (10.2)* 14 (9.2)* 
Pills 3 (1.8)* 4 (2.6)* 
Injectable 2 (1.2)* 8 (5.3)* 
Abstinence 7 (4.2)* 3 (2.0)* 
Male sterilization 2 (1.2)* 3 (2.0)* 
Female sterilization 1 (0.6)* 0* 
Traditional 4 (2.4)* 2 (1.3)* 
IUD 0 0 
Diaphragm 0 0 
LAM 0 0 
Foam or Jelly 0 0 
Emergency contraception | 0 0 


"Multiple responses 


3.5 Factors Affecting Uptake of Contraception Among HIV Positive Male Patient 


3096 of the respondents choose that the reason for not using contraception was because 
they were married, 29% said it is because they were both HIV positive, 28% said it 
reduces sexual pleasure, 18% acknowledge that their religion does not permit them to 
use it also that they don't know where to obtain contraception from and lack of discussion 
by their health care provider. 1596 reported that it was because of the side effects, 1496 
due to the attitude of the health workers, 7% reported that it was because contraception is 
too expensive. Out of 10% of the respondents that were not using contraception because 
of other reasons, most (13) of the respondents reported that it was because they are not 
married. 


3.6 Sociodemographic Characteristics, Knowledge, Perception and Uptake 
of Contraception Relationship 


The relationship between the demographic characteristics and the knowledge, perception 
and uptake of the patients were evaluated as shown in Table 3. Findings from the study 
indicated all demographic characteristics were found to be insignificant except for marital 
status which shows good relation with the knowledge (P < 0.001) and uptake (P « 0.001); 
including other characteristic like Level of education which was shows a significant 
relationship with their perception (P — 0.005) and uptake (P — 0.004) while age was 
found to have shown as significant influence (P — 0.001) on their use of contraception. 

The study further indicates that the most influential demographic characteristic which 
significantly contributes to the level of the patients’ perception, knowledge and uptake 
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Table 3. Relationship between sociodemographic characteristics and other studied criteria. 


Socio-demographics | Relationship X? (P-value) 
Yes % No % 

Marital status Socio-demographic-Knowledge 

Single 28 (68.3) 13 (31.7) | 25.300 (<0.001)* 

Married 190 (89.6) 22 (10.4) 

Widowed 6 (100.0) 0 (0.0) 

Divorced 4 (44.4) 5 (55.6) 

Level of education Socio-demographic-Perception 

No formal education | 5 (62.5) 3 (37.5) 16.676 (0.005)* 

Primary 27 (67.5) 13 (32.5) 

Quranic education 39 (58.2) 28 (41.8) 

Secondary 52 (71.2) 21 (28.8) 

Post-secondary 20 (76.9) 6 (23.1) 

Tertiary 49 (90.7) 5 (9.3) 

Age (years) Socio-demographic & use of contraception 

18-27 2 (12.5) 14 (87.5) | 23.686 (0.001)* 

28-37 23 (46.9) 26 (53.1) 

38-47 61 (72.6) 23 (27.4) 

48-57 42 (53.9) 36 (46.2) 

58-67 20 (58.8) 14 (41.2) 

68-77 3 (60.0) 2 (40.0) 

78-87 1 (50.0) 1 (50.0) 

Marital status 

Single 12 (29.3) 29 (70.7) |17.958 («0.001)* 

Married 133 (62.7) 79 (37.3) 

Widowed 4 (66.7) 2 (33.3) 

Divorced 3 (33.3) 6 (66.7) 

Level of education 

No formal education |1 (12.5) 7 (87.5) 17.544 (0.004)* 

Primary 19 (47.5) 21 (52.5) 

Quranic education 33 (49.3) 34 (50.8) 

Secondary 43 (58.9) 30 (41.1) 

Post-secondary 15 (57.7) 11 (42.3) 

Tertiary 41 (75.9) 13 (24.1) 
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of contraception was found to be marital status, education level and age. Which were 
found to be evident in the nature of the responses where the married class and aged class 
of people (from 38 years and above) were found to have shown significant knowledge of 
contraception compared to other categories. The influence of education was equally seen 
to have enabled the patients’ gain better perception of the contraception use compared 
to the class of uneducated patients. 


4 Conclusion and Recommendations 


The study carried out in NTBLTC Saye-Zaria to determine the knowledge of contra- 
ception among HIV positive male patients showed that there was high knowledge of 
contraception among HIV positive men, good perception, low usage of contraception 
and poor acceptance of contraception due to religious reasons. Marital status, level 
of education, and age were socio-demographic characteristics that affected the uptake 
perception and knowledge of contraception. Therefore, it is necessary for religious orga- 
nization to enlighten men on importance of contraception. The Government should put 
in place adequate policies to encourage male involvement in the utilization of contra- 
ception as some respondents in the study thought contraception is basically a problem 
the women have to deal with alone. 
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Abstract. This paper presents AcousticPAD, a contactless and robust 
handwriting recognition system that extends the input and interactions 
beyond the touchscreen using acoustic signals, thus very useful under the 
impact of the COVID-19 epidemic. To achieve this, we carefully exploit 
acoustic pulse signals with high accuracy of time of fight ((ToF) measure- 
ments. Then we employ trilateration localization method to capture the 
trajectory of handwriting in air. After that, we incorporate a data aug- 
mentation module to enhance the handwriting recognition performance. 
Finally, we customize a back propagation neural network that leverages 
augmented image dataset to train a model and recognize the acoustic sys- 
tem generated handwriting characters. We implement AcousticPAD pro- 
totype using cheap commodity acoustic sensors, and conduct extensive 
real environment experiments to evaluate its performance. The results 
validate the robustness of AcousticPAD, and show that it supports 10 
digits and 26 English letters recognition at high accuracies. 


Keywords: Acoustic sensing - Gesture recognition - Wireless sensing 


1 Introduction 


Nowadays, touchscreen technology [1] has been widely used as a way to interact 
with computer systems. For example, we often use the self-ordering screen in 
the KFC and McDonald as shown in Fig. 1(a). In Automated Teller Machines 
(ATMs) information kiosks, we use touchscreen to input password and withdraw 
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cash. However, due to the world is being affected by COVID-19, and this contact- 
type interaction method will increase the spread of the disease [2]. Imagine that 
if there is a contact-free interaction method, it will greatly reduce the spread of 
infections. In this paper, we aim to design a robust contact-free sensing system 
that can leverage cheap acoustic sensors to achieve accurate and robust input 
with handwriting in the air. 


— 


(a) Traditional touchscreen (b) Contactless handwriting 


Fig. 1. The evolution from traditional touchscreen to our proposed contactless acoustic- 
based handwriting system 


Some existing contact-free handwriting systems require extra customized 
devices with high cost (e.g., FMCW radar [3,4] and Lidar [5,6]). Some work [7- 
10] utilizes WiFi signal which has been proved with severe location-dependent 
issues. Recent work [11-13] utilizes acoustic signals embedded in smart devices 
to enable gesture tracking. FingerIO [11] can accurately track waving hand by 
transmitting OFDM (Orthogonal Frequency Division Multiplexing) modulated 
acoustic signals. CAT [14] utilizes acoustic FMCW (Frequency Modulated Con- 
tinuous Waveform) to develop fine-grained motion tracking systems. However, 
these systems should tackle complex system delay and sampling frequency offset 
before use. And the distance measurements are difficult to obtain accurately. 

In this paper, we propose AcousticPAD, a contact-free handwriting recog- 
nition system based on cheap commodity acoustic sensors. In our solution, we 
place two acoustic sensors on the corner of a surface and transmit acoustic pulse 
signals (Fig. 1(b)). By simply setting the signal voltage, the system can accu- 
rately measure the echo pulse reflected from user's hand and further estimate 
the flight time of the signal. Combined with ToF from two acoustic devices, we 
track the hand trajectory using the trilateration localization method. To recog- 
nize the content of handwriting input, we need to have the classification method 
and address the challenges of labor-intensive data collection and handwriting at 
differen locations and orientations in the air. Therefore, we borrow the existing 
dataset (MINIST) [15] from image recognition field and design a data augmen- 
tation technique to enhance the data. Such a well-designed recognition process 
can not only reduce the time and effort required to manually collect data but 
also achieve the location and orientation independent handwriting recognition. 
Experiment results demonstrate our system is able to recognize handwriting of 
10 numbers and 26 English letters with high accuracy and robustness. Please 
find our demo video at the link: https://youtu.be/sCZvK2rUzEU. 


In-Air Handwriting Recognition Using Acoustic Impulse Signals 295 


The main contributions of the paper are summarized as follows: 


— We propose a novel contactless handwriting recognition approach, which 
enables surface-drawn interfaces using acoustic pulse signals. Compared with 
existing approaches employing FMCW [16] or OFDM [11] acoustic signals, 
the proposed pulse acoustic signals have the advantages of accurate position- 
ing and low energy consumption. 

— We develop a series of signal processing techniques to realize the system. 
Integrated with existing MINIST dataset and proposed data augmentation 
method, we are the first to demonstrate the possibility of using cross-domain 
training in a contactless sensing system. 

— We implement a prototype handwriting recognition system using commodity 
cheap acoustic devices and conduct evaluations. Evaluation results show that 
our system is robust against writing location and orientation and the average 
recognition accuracy of 10 digits and 26 letters is greater than 90%. 


2 System Design of AcousticPAD 


2.1 System Overview 


Figure2 illustrates the overview about the design of the proposed AcousticPA D 
system that leverages commercial acoustic sensors to transmit/receive pulse 
signal. AcousticPAD mainly consists of two modules: Real-Time Handwriting 
Acquisition and Position-Independent Handwriting Recognition. In Real-Time 
Handwriting Acquisition module, AcousticPAD collects sound signal from acous- 
tic sensor and recovers the trajectory of handwriting. Then Position-Independent 
Handwriting Recognition module leverages the augmented MNIST/EMNIST 
dataset [15] to train a Back Propagation (BP) neural network and recognize 
the handwritings including digits and letters position-independently!. 


Scenario Real-Time Handwriting 
Acquisition 
Tigger to Transmit and Receive 
Sound Pulse Signal 


m Distance Measurement 
Real-Time Segmentation 


Je Handwriting Tracking 


Ultrasonic Device 
| HC-SR04 


Character Smoothing Filter 


Position-Independent Handwriting Recognition b 


nds (3)—Digits/Letters Output] — Classification 


BP Network Training 
— M 


Offline Original Data 
phase J|MNIST/EMNIST| | Augmentation 
Dataset i n 
Fig. 2. System overview of AcousticPAD. Fig.3. The trilateration localiza- 


tion method for handwriting. 


1 Position independent refers to the user can perform the handwriting without location 
and orientation dependency. 
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2.2 Real-Time Handwriting Acquisition 


In this subsection, we first introduce the generation of acoustic pulse signal 
in our system. Then we acquire the handwriting characters in a contactless 
manner following steps of: distance measurement, character segmentation and 
hand tracking. 


Trigger to Transmit and Receive Sound Pulse Signal. In AcousticPAD, 
two HC-SR04 acoustic sensors [17] are employed to transmit and receive sound 
pulse signal. The sensor contains four pins, i.e., VCC, TRIG, ECHO and GND. 
All the pins are connected to the Raspberry Pi [18]. When AcousticPAD works, 
Raspberry Pi supplies 5 V and 0 V voltage to the VCC and GND pins of sensor, 
respectively. Once a TTL (Transistor-Transistor Logic) pulse signal that lasts at 
least 10 us is sent to the TRIG pin, the sensor automatically transmits 8 pulses 
signal at 40 KHz frequency and raises ECHO pin from low-level voltage to high- 
level voltage. Then the sensor monitors the echo-back signal. If the echo-back 
signal is received, the sensor converts the ECHO pin from high-level voltage to 
low-level voltage. Thus the time duration of high-level voltage in ECHO pin is 
the time of flight (ToF) for ultrasonic signal. 


Distance Measurement. Suppose that the time for transition of the ECHO 
pin voltage is T, (from low-level to high-level) and Ty (from high-level to low- 
level), the ToF of ultrasonic signal is the difference between T, and Ty. Thus the 
distance from the sensor to user’s hand can be denoted as: 
(Te — T, 
d= c ( f ) (1) 
2 

where c is the speed of ultrasound signal in the air. For both acoustic sensors, 
the distances can be measured as the input of subsequent steps. 


Real-Time Character Segmentation. With accurate distance measure- 
ments, we segment the consecutive handwriting inputs in realtime. As shown in 
Fig. 3, when the user's hand appears in the sensing area, the measured distance 
from two sensors denoted as dı and d» will decrease. This is because without 
sensing target, the sensors measure a long distance in the environment that are 
outside of the sensing area. While completing the character input, the measured 
distance increases after user's hand gets away from the sensing area. Thus we 
can set a threshold based on the sensing area to detect whether user's hand in 
the sensing area to segment the handwriting in realtime. When the distances 
of both acoustic sensors are less then a specific threshold, AcousticPAD starts 
to localize user's hand and track handwriting trajectory. Otherwise, the user 
finishes his input and AcousticPA D outputs the tracking results and feeds it to 
the recognition module. 
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Hand Tracking. We design a trilateration localization approach to localize 
user’s hand and track handwriting trajectory. As is shown in Fig. 3, two acoustic 
sensors are deployed at the corners of the input panel (e.g., table surface). The 
distance s between two sensors is set in advance. With distance measurements 
from hand to sensors, the three edges of triangle OAB constituted by sensors 
and user’s hand are known. According to the law of cosines [19], the angle of 
user’s hand @ with respect to OA satisfies: 


2 d2 = d2 
0 = arccos Uo (2) 
Thus the postiion of user's hand is: 
x = d,cosé 
{ y = disin0 (3) 


For each distance measurement, AcousticPAD can obtain the position of user's 
hand. After successively connecting all the discrete positions during input, we are 
able to aquire handwriting performed by the user. Before feeding the handwriting 
into recognition module, we employs Savitzky-Golay filter [20] to smooth the 
handwriting. The filter utilizes linear least square method to fit successive subset 
of adjacent positions with a polynomial. After filtering out the noise, we obtain 
the final handwriting trajectory as the input of recognition module. 


2.8 Position-Independent Handwriting Recognition 


To achieve the position-independent handwriting recognition, we divide the 
recognition module into offline phase to train a cross-domain model with datasets 
from exisiting imaging dataset and online recognition with contactless acoutic 
system generated handwriting dataset. 


Offline Phase: Different from existing work that needs to collect a large num- 
ber of samples to build dataset and train a classification model, Acoustic PAD 
leverages existing handwriting datasets, i.e., MNIST and EMNIST [15], as the 
training datasets without using acoustic sensing system generated data, which 
requires zero data collection effort. We notice that the data samples in MNIST 
are specific with certain style. If we directly utilize the original datasets to train 
a model, the model is position dependent with low generalization ability. To 
solve this problem, we propose a data augmentation approach to enhance the 
original datasets with wide applicability. Specifically, we first transform the char- 
acter image samples with different rotation angles. Assume that the orientation 
of original image samples is 0°, the images are duplicated, converted from 0? to 
90? (anticlockwise) and —90? (clockwise) with a step size of 15?. Secondly, we 
move the image samples in both vertical and horizontal directions with a step 
size of 14 pixels. After these operations, the augmented dataset is 65x larger 
than the original ones and contains image samples in different positions. 

As shown in Fig. 2, we utilize the augmented dataset to train a three-layer BP 
neural network, which includes a input layer, a hidden layer and an output layer. 
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Due to the size of images is 28 x 28 pixels, we convert them into a vector 1 x 784 as 
input. Thus the number of nodes in the input layer is 784. And we set the number 
of nodes in the hidden layer as 500. The number of nodes in the output layer is 
10/26 for digit/letter recognition, respectively. During the training process, the 
image data are fed in the forward direction by propagating from the input layer 
through the hidden layer to the output layer, we can calculate the error between 
output value and expected value of output layer. Then the backward propagation 
of errors are applied to modify the connected weight values with learning rate 
0.1. In our cases, all the samples in the augmented datasets are used to train 
the model, while the data collected by AcousticPAD system are for testing. 
Benefiting from the data augmentation, the trained model is robust against 
position changes and can achieve position-independent handwriting recognition. 


Online Phase: For online phase, AcousticPAD takes the acoustic sensing sys- 
tem generated handwriting trajectory as input and leverages the model trained 
in offline phase to classify the handwriting characters. AcousticPAD employs a 
camera to record the real-time video of user’s handwriting as ground truth. We 
develop a web based front end. After the user performs handwriting, the cor- 
responding recognition result is displayed on the web page. AcousticPAD auto- 
matically segments the successive characters so that the user can continuously 
input numbers or letters using this system. 


3 Evaluation 


3.1 Experiment Setup 


We have implemented AcousticPAD with two commodity cheap acoustic sen- 
sors (i.e., HC-SR04) [17] (5$ per unit), and a Raspberry Pi 4B module [18]. The 
acoustic sensors transmit/receive acoustic pulse signals with 40 KHz. The oper- 
ation voltage and current are DC 5 V and 15 mA, respectively. The trigger input 
signal is 10 us TTL pulse. We develop a web-based user interface to demonstrate 
the handwriting recognition in real time, as shown in Fig. 4. The demo video can 
be found at: https://youtu.be/sCZvK2rUzEU. 


0[234796789AB 
CDOZFGHIIJKLMN 
OPQRSTUVWXYZ 


Fig. 4. Testbed and experiment scenario. Fig. 5. Hand tracking results of 10 
digits and 26 letters. 
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3.2 Performance Evaluation 


Hand Tracking. We first evaluate the hand tracking results. The volunteer 
performs hand gestures including 10 digits (0-9) and 26 letters (A-Z) in front of 
a table, and two small acoustic devices are placed at the table corners. As shown 
in Fig.5, we show examples of all the digit and letter trajectories captured by 
our contactless handwriting system. It can be seen that our system is able to 
track the details of hand positions for writing all the digits and letters. With 
different angles of straight-line strokes and curves, the obtained trajectories can 
perfectly match the shapes of the characters. 


Character Recognition Accuracy. Then we evaluate the character recog- 
nition accuracy. We let 10 participants perform handwriting experiments. For 
each character, the participants repeat 3 times with 5 different positions and 
5 orientations, which construct a sample set with 27,000 samples in total (10 
participants and 36 characters). We use the data augmented MNISIT dataset 
to train the BP neural network. All the acoustic sensing system collected data 
are used to test the recognition performance. Figure 6 illustrates the confusion 
matrix achieved by AcousticPAD for each character. The overall recognition 
accuracy for digits and letters are 92% and 90.3%, respectively. While the accu- 
racy of some characters is relatively low (about 87%), for example, ‘3’, ‘Z’, this 
might be because the users perform ‘3’ and ‘Z’ with some irregular handwriting 
style which makes easier to confuse with other characters. 


0 0.01 0.03 M 09 4B T 
B i 
1 0.010.010.030.03 08 p ik 
F 
2 03 0.01 0.04 0.04) Jo7 Gt » 
n I 
4 3 .020.010.04 0.08 | o6 2 dL " 
ym — 01 
O4 003 0.04) Jos 5 Y . os 
25 0.03 0.010.041 m 
E 104 gor. 04 
5 03 E b. - 
7 0.07 pj "Um a 
0.2 U p 07 001001. 02 
8 0.020.010.01 XEM ws " E ME 
0.1 X o oo ow — om 0.1 
9 0.03 0.01 M d "X de "a J 
Z opt , apt opr opt, apa 0 
0123 4 5678 9 ABCDEFGHI JKLMNOPQRS TUVWXYZ 
Predicted Class Predicted Class 
(a) Digits (b) Letters 


Fig. 6. Confusion matrix of recognition accuracy 


Position Independent Recognition. To demonstrate the position indepen- 
dent capability, we conduct the handwriting of all characters at 5 different 
locations and 5 orientations, and compare the performance with BP neutral 
model [21] without data augmentation. Figure 7 shows the comparison results of 
accuracy for two methods. Our methods achieve above 8696 accuracy for both 
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five locations and orientations. Compared with the model without data augmen- 
tation, our method improves accuracy for more than 32% on average, which also 
demonstrates the robustness of AcousticPAD system. 
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Fig. 7. Recognition accuracy at different positions 


Conclusion 


In this paper, we propose AcousticPAD, a contactless handwriting recognition 
system that employs acoustic pulse signals to enable input interactions with 
cheap acoustic sensors. AcousticPAD captures acoustic signals when hand moves 
on the table surface, and then track the handwriting trajectory to recognize 


the 


characters. We have implemented AcousticPAD as a real-time recognition 


system and conducted comprehensive experiments to validate the effectiveness 
and robustness of the system. Our results demonstrate high input recognition 
accuracies across different users with position-independent ability. 
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Abstract. The presented demonstrated working tools in the initial version con- 
stitute the foundation of the novel ALS and MS management and monitoring, 
leveraging extended IoT sensing and emerging instruments infrastructure, and a 
basis for integration of more advanced and effective AI models (in development) 
for disease progression prediction, patient stratification and ambiental exposure 
assessment. 
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IoT integration 


1 Introduction and Scientific/Technological Background 


The BRAINTEASER Project (BRinging Artificial INTElligence home for a better care 
of Amyotrophic lateral Sclerosis and multiple scIERosis), funded from the European 
Commission Horizon 2020 programme grant number GA101017598 until 2024, aims to 
integrate societal, environmental and health data to develop patient stratification and dis- 
ease progression models for Amyotrophic Lateral Sclerosis (ALS) and Multiple Sclerosis 
(MS). ALS and MS are both very complex chronic progressively degenerative neurolog- 
ical diseases significantly disrupting the quality of life of the patients and their families, 
but with very different clinical picture, evolution, prognosis and therapies. 

BRAINTEASER integrates detailed retrospective and prospective clinical 
datasets with comprehensive heterogeneous personal health, activity, lifestyle, habit- 
ual/behavioural, and environmental data collected using commonly available sens- 
ing/IoT devices and the demonstrated first release of developed interactive tools for 
disease monitoring and management. 


© The Author(s) 2022 
H. Aloulou et al. (Eds.): ICOST 2022, LNCS 13287, pp. 302-310, 2022. 
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The collected data drive the development of Artificial Intelligence (AI) models able 
to address the needs of precision medicine, enabling early risk prediction of disease 
fast progression and adverse events, and the next planned releases of the tools currently 
in development will focus on the integration of the AI models and their outputs into 
generated specific highly complex feedbacks to the users (digital gamified coaching and 
recommendations to improve the patient’s and caregiver’s daily routines and overall 
well-being, and the analytics suite for full patient cohorts management according to 
stratifications output of the AI models and for interactive visual data exploration and 
detailed understanding of the disease progress and evolution of patient’s conditions). 

The demonstrated tools are being devised and developed embracing an agile user- 
centered multidisciplinary co-design approach, accounting for the specific technical, 
medical (including psychological/cognitive) and societal needs of the users. Proof-of- 
concept of their validated use in real clinical settings in 4 clinical study validation sites 
(Lisbon, Madrid, Pavia & Turin) aims to provide quantitative evidence of benefits and 
effectiveness of leveraging AI models in healthcare pathways for dire neurodegenerative 
diseases, and a set of recommendations for the health authorities to support the transition 
of the current approach to healthcare from reactive to predictive, paving the path towards 
a healthier and more fulfilling life as long as possible. 


* * 
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Fig. 1. Main data streams across the high-level BRAINTEASER ecosystem organization. 


The presented tools are primary end-user-oriented components of the overall 
BRAINTEASER platform ecosystem, supported by the underlying unified Data Plat- 
form, Semantic Cloud, middle tier of integration service APIs, and others as presented 
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on Fig. | above with relevant main data streams and the involved external relations, 
including Project Open Science and Evaluation Challenge actions already underway. 
All together in turn provide support for the identified processes enhancing the man- 
agement of ALS and MS both throughout the Project demonstration study and the care 
provision practice beyond, differentiated and structured through comprehensive analysis 
and co-design with relevant stakeholders as presented on Fig. 2 below, and progressing 
beyond the thoroughly analyzed key features of current relevant state-of-the-art ALS & 
MS management tools and applications like Emilyn!, Cleo”, and over 50 others relevant 
analyzed and compared in terms of specific features and functionalities. 


Clinical process 


Selection Initial Personaliza Devices 
Eas Enrolment 7 à 
criteria assessment tion config 


Brainteaser core 
End of Hospitalization Clinical Early Remote 
service [other events follow up detection monitoring 
2b 
4. Patient activities 
First use at Patient Digital Remote 
Feedback 
home monitoring interventions updates 


Fig. 2. Overall BRAINTEASER process flows enhancing disease management of ALS and MS. 


Some of the platform functionalities and architecture build and extend upon preced- 
ing and parallel related projects and initiatives tackling common issues and exploiting 
applicable methods in neurodegenerative progression management [7], well-being [8], 
functional decline prevention [9], and public health [4, 5], such as PULSE’, City4Age*, 
NEVERMIND’, or VERITAS®. 

One of the set out goals of BRAINTEASER is also to analyze the possible effects 
and influences of environmental and daily living ambient factors (primarily air pollution) 
on the aetiology and progression of neurodegenerative sclerotic diseases like MS, as has 
been intensely studied in recent years [1—3] along with water and soil pollution, and the 
role of these factors on progression and relapse of MS and ALS, feeding the relevant col- 
lected environmental data into the dedicated developed AI models for patient exposure 
to the ambiental factors, complementing the models for disease progression and stratifi- 
cation. The cities with clinical trial sites involved in the project (Lisbon, Madrid, Pavia, 


1 https://play.google.com/store/apps/details?id=com.bxh_mvp&hl=es. 
2 https://www.cleo-app.es/index.html. 

https://www.project-pulse.eu. 

i https://cordis.europa.eu/project/id/68973 1. 

3 https://www.nevermindproject.org. 

5 https://cordis.europa.eu/project/id/247765. 
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Torino) have established extensive environmental sensing and measuring infrastructure, 
ingesting data both from public environmental measuring stations wherever open or 
available, as well as from connected IoT sensing devices deployed by BRAINTEASER 
and previous or parallel relevant projects like (again) PULSE, EMOTIONAL Cities’, 
Sharing Cities’, ACROSSING? and others [4-6], and complemented also with data 
from portable/wearable compact sensing devices (carried on or with patients) deployed 
by the project, mostly Atmotube PRO monitors!?. 


2 Interactive Disease Management and Monitoring Tools Features 


Three main interactive tools are being developed to support the processes illustrated 
on Fig. 2 above, differentiated and tailored towards specific target end-user roles and 
distinctly siloed functionality groups required for each group of roles: 


]. Native mobile app. for patients and informal caregivers (family members, relatives, 
home nursing/assistance...) 

2. Clinical Tool web application for specialist clinicians and/or physicians 

3. Data Management and Exploration (Dashboard-based) Tool, a hybrid application 
for clinical study managers or administrators, care provision executives or policy- 
makers, and researchers to explore data and trends from selected studied patient 
cohorts down to individual levels, where meaningful and not obfuscated or unavail- 
able due to indirect anonymization necessary for personal data protection purposes, 
and experiment with AI models execution and parameter tweaking/fine-tuning. 


In this initial release, the mobile app. for patients/caregivers and the Clinical web 
Tool are primary, to support the key required functionalities of the Project study, enrolling 
the patients and: 


e quantitative and qualitative information collection, consolidation and fusion (of het- 
erogeneous data from IoT wearable trackers and environmental sensing networks, 
combined with data from digitalized standardized and innovative evolving instru- 
ments and questionnaires for ALS and MS (and comorbidities) clinical evaluation 
and remote disease progress assessment), 

e elementary multimodal disease management, monitoring and assistance in daily 
patient and caregiver needs in remote and clinical settings, enabling follow-up, symp- 
toms and health issues resolving, and feedback from the clinicians and caregivers to 
the patient through interactive GUI types, 


as diagrammed on Fig. 3 below. 
Next subsequent version in current development is focused on key “intervention- 
related" features listed on bottom left and bottom right of Fig. 3, providing support to 


7 https://emotionalcities-h2020.eu. 

8 www.sharingcities.eu 

9 www.acrossing-itn.eu 

10 https://atmotube.com/atmotube-pro. 


306 S. Gonzalez-Martinez et al. 


patients, caregivers and clinicians to fully manage, prevent and mitigate disease progres- 
sion, from basic advisory guidance and instructions to advanced cognitive games and 
exercises, or voice/speech assessment in evolution of bulbar symptoms. 


Patient APP 
E NE aa Clinical Tools 
~ A [ LE 
e» 
2: 
@ Myhealth . + 
$ Activities H - 
Q contact Ee 
$ . 
o - 
Questionnaire - Risk assessment tools 
Physiological / activity monitor Exposure to air pollutants - Prognostic tools 
Support: Environmental risks - Remote management 
- Educational materials - Assessment procedures 
- Contact with health professionals - Relapse management 
- Rehabilitation exercises 
- Mindfulness Environmental 
- Cognitive games sensors 


Fig. 3. Conceptual schema of primary functionalities of the BRAINTEASER Patient App. and 
Clinical Tool, with key data flows. 


Example screen form for support service configuration in the MS-specific variant of 
the Clinical Tool is provided on Fig. 4 below and the detailed breakdown of UML use 
cases notation of basic functionalities of the Patient App. on Fig. 5 below. 
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Fig. 4. Clinical Tool screen example for MS service configuration. 
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Fig. 5. UML notation of use cases covering key functionalities of the Patient/caregiver app. 


The third listed Interactive Data Management and Exploration (dashboard-based) 
Tool is in initial version development as it significantly depends on and is intended 
to integrate substantial output from the AI models which are still in development and 
testing, but at this stage provides at least rich interactive visual exploration of arbitrary 
selection of collected temporal and heterogeneous data variables (including categorized 
and continuous), with values evaluated or compared combined together in multimodal 
and multidimensional composite 2D and 3D visualizations, as exemplified on Fig. 6 


below. 
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Fig. 6. Multivariate composite 3D interactive exploratory data visualization of 7 different com- 
bined continuous and categorized variables, additionally stratified per gender and marital status 
demographics. 


3 Key Architectural and Service Design Challenges 


The tools are designed to be modular, dynamically adaptable, personalized and support- 
ing alternative user interaction modalities (with significantly higher accessibility than 
common for “ordinary” apps, adapting to the increasing manual dexterity impairments 
of the patients), with simple and scalable navigation structure, optimal balance between 
menu structure and dynamic interaction flows, and designed for long-term usage, in the 
UI/UX aspects. 


Wearable Device 
A device 
vendor API =. 
> Environmental device 
vendor API 


Sensing 
Daves —- 
= S 
med E 


Brainteaser back-end services 


Brainteaser Semantic Data Cloud 
Retrospective Clinical Data. 


n Data Source Li Data Consumer Oo Data Source/Consumer | 


Brainteaser Data Storage 


Fig. 7. BRAINTEASER ecosystem architecture with main data sources, consumers and flows, 
and detailed service tier breakdown. 


In the overall architecture of the underlying supporting BRAINTEASER data collec- 
tion, management, and provision ecosystem (Fig. 7 above), similar microservice modu- 
larity methodological principles have been followed, including the separation between 
domain-specific and infrastructural/utility orthogonal logic, and taking into account the 
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system and service performance and security aspects built into the design and archi- 
tecture from the start, employing loose coupling and service orchestration patterns like 
the baseline API Gateway facade pattern that may evolve into more comprehensive 
orchestration pattern (like ESB) if complexity or performance criticality increases in the 
ongoing iterative further advancement of the architecture. 


4 Conclusion 


The presented demonstrated working tools in the initial version constitute the foundation 
of the novel ALS and MS management and monitoring leveraging extended IoT sensing 
and emerging instruments infrastructure, and a basis for integration of more advanced 
and effective AI models (in development) for disease progression prediction, patient 
stratification and ambiental exposure assessment. 
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